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FOREWORD 


Historically,  Etectronic  Warfare  (EW)  Systems  have  been  complex  and 
frequently  prewnt  a  dichounny  because  they  represent  high  technology  at  its  best  and 
its  worst.  At  its  best,  EW  can  be  a  major  technical  and  tacdcal  cdgic  (force>mult4>lier) 
over  potential  adversaries  (as  demonsnated  during  recent  military  ofterations);  \riiile  at 
its  worst,  EW  can  incur  vicious  spiraling  costs  and  can  push  risk  and  technologkal 
state-of-the-art  to  keep  pace  widi,  or  exceed,  advances  in  potential  threat 

systems  and  their  counter-countermeasures  (CCM).  Needs  for  new  EW  capabilities 
arise  usually  in  response  to  known  or  erqpected  actions  by  a  potentially  hostile  entity 
about  which  little  may  be  known.  The  need  for  rapid,  decisive  response  can  cause  the 
acquisition  process  for  new  EW  capabilities  to  be  less  conventional  and  more  dynamic 
than  other  types  of  major  systems.  As  development  proceeds,  additional  intell^ence 
may  become  available  that  may  cause  abrupt  changes,  or  as  a  minimum  a  re-analysis  of 
the  requirements,  i.e.,  has  the  threat  changed,  expanded,  or  increased  in  complexity 
enough  to  make  our  new  capability  ineffective  when  fielded?  Because  EW  is  so 
dependem  upon  the  use  of  current,  and  in  many  cases,  highly  sensitive  intelligence 
sources  and  methods,  the  openness  of  information  relating  to  new  EW  capabilities  must 
carry  the  same  level  of  protection  to  prevem  revealing  details  of  the  capability. 

EW  systems,  to  be  effective,  have  to  be  flexible  in  responding  to  a  multiple- 
threat,  muitispectral  environment  which,  in  turn,  leads  to  complex  EW  systems. 

Trends  indicate  that  current  and  future  EW  systems  will  be  required  to  be  integrated 
with  or  carried  by  a  variety  of  platforms.  New  capabilities  must  respond  in  as  close  to 
real-time  (nanoseconds)  as  technically  and  operationally  feasible.  T^  infers  that  the 
systems  must  keep  pace  with  the  state-of-tle-art  in  not  only  EW  technologies,  but  also 
in  the  technologies  they  must  counter,  e.g.,  low-probability-of-intercept,  low 
observables,  high-power  microwaves. 

EW  systems  should  bring  added  value  to  the  warfighter.  The  effectiveness  of 
EW  is  difficult  to  measure  but  can  be  tied  directly  to  the  survivability  of  the  assets  it  is 
intended  to  protect.  It  should  be  as  non-intrusive  as  possible  on  the  warfightii^  itself. 
Because  the  art  of  EW  has  some  unconventional  characteristics  and  demands,  its 
conformance  to  accepted  acquisition  practices  has  often  been  outside  the  perceived 
norm.  However,  from  a  T&E  process  point  of  view,  EW  systems  development  is  not 
so  different  or  special  that,  with  the  proper  precautions  in  place  to  prevent  premature 
revelation  of  capability,  it  cannot  be  adapted  to  a  standardized  DoD  T4&E  process.  The 
T&E  process  for  EW  systems  described  in  this  document  represents  a  compendium  of 
best  T&E  practices  and  procedures  and  is  designed  to  allow  that  conformance  to  be 
achieved. 
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A  DESCRIPTION  OF  THE  DOD  TEST  AND  EVALUATION  PROCESS 

FOR 

ELECTRONIC  WARFARE  SYSTEMS 


1.  INTRODUCTION 

1.1  Objecdre.  The  principal  objective  of  this  doaunent  is  to  describe  a 
rigorous,  standardized,  disciplined  test  and  evaluation  (T&E)  process  for  Electronic 
Warfare  (EW)  systems.  The  Process  is  comprised  of  functions  and  stq>s  to  assist  the 
decision  making  process  leadii^  to  progressive  accpiisition  of  mission-required  EW 
systems. 

1.2  Badcground. 

1.2.1  In  mid-1993,  the  Under  S«netary  of  Defense  (Acquisition  & 
Technology),  Director,  Test  and  Evaluation  convened  a  task  force  to  develop  a  process 
for  EW  systems  T&E.  This  task  force,  with  the  concurrence  of  their  Service 
Executives,  produced  a  DoD  T&E  Process  for  EW  systems  described  in  this  document. 
DoD  Instruction  50(X},2,  Defense  Acquisition  Policies  and  Procedures,  was  used  as  the 
baseline  for  this  process.  That  is,  DoDI  5000.2  provides  the  friunework  around  this 
DoO  T&E  Process  for  EW  systems.  This  document  supports  DoDI  5000.2  and  is  not 
in  conflict  with  the  5000  series  of  documents. 

1.2.2  Specific  program  repordi^  requirements  are  announced  separately 
by  DT&E  aiKi  OT&E.  The  T&E  process  described  herein  may  be  applied  to  EW 
systems  that  conform  to  the  definition  in  Paragraph  1.3.1  of  this  document.  This 
definition  is  from  CJCS  Memorandum  of  Policy  (MOP)  No.  30  that  promulgates 
Command  and  Control  Warfare  (C2W)  as  a  military  strategy.  In  addition  to 
establishing  C2W  as  a  new  strategy,  MOP  30  effectively  eliminates  Command, 

Control,  Communications  Countermeasures  (C3CM)  as  a  military  philosophy  and  a 
working  definition  by  replacing  it  with  C2W  (see  Paragraph  1.3.2).  Two  other 
changes  promulgated  by  MOP  30  in  the  shift  from  C3CM  to  C2W  are:  1)  to  change  the 
term  "jamming’*  to  the  much  broader  military  action  of  EW  and  2)  to  add  psychological 
operations  (PSYOP)  as  a  principal  military  action.  The  latter  (along  with  operational 
security  (OPSEC))  is  not  addressed  in  this  document. 

1.3  Terminology. 

1.3.1  Electronic  Warfare  (EW).  Electronic  Warfare  is  a  military  action 
involving:  1)  the  use  of  electromagnetic  or  directed  energy  to  attack  an  enemy's 
combat  capability,  2)  protection  of  friendly  combat  capability  against  undesirable 
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effects  of  friendly  or  enemy  employment  of  EW,  or  3)  surveillance  of  tbe 
electromagni^  spectrum  (also  known  as  EW  Stpport)  for  immediate  threat  recognition 
in  support  of  electronic  warhire  operations  and  other  tactical  actions  such  as  threat 
avoidance,  targetii^,  and  homing. 

1 .3  ■  1  ■  1  Electronic  Attack  CEAV  Involves  the  use  Of 
electromagnetic  or  directed  energy  to  attack  personnel,  facilities,  or  equipment  with  the 
intent  of  degrading,  neutralizing,  or  destroying  enemy  combat  capabili^. 

Representative  examples  include,  but  ate  not  limited  to:  jamming,  e.g.,  noise, 
rqieater;  decqition  countermeasures  (DECM),  e.g.,  frdse  targets,  range  denial,  velociQr 
(Doppler)  denial;  physical  kill,  e.g.,  high-power  microwaves. 

1 .3. 1.2  Electronic  Protection  <EP).  Involves  the  actions  taken  to 
protect  personnel,  facilities,  and  equipment  from  any  effects  of  friendly  or  enemy 
employment  of  EW  that  degrade,  neutralize  or  destroy  friendly  combat  capability. 
Representative  examples  include,  but  are  not  limimd  to:  automatic  warning  sensors, 
e.g.  radar  homing  and  wamiip  (RHAW),  missile  attack  warning,  laser  warning; 
decoys/RPVs;  expendables,  e.g.,  chaff,  flares  (IRCM);  electronic  counter- 
countermeasures  (ECCM),  e.g.,  antijam  techniques,  radar-cross-section  (RCS) 
reduction,  wartime  reserve  modes  (WARM),  and  anti-air,  anti-radiation  missile 
(AARM), 


1.3. 1.3  Electronic  Support  (ESV  Involves  actions  tasked  by,  or 
under  direct  control  of,  an  operational  commander  to  search  for,  intercept,  identify, 
and  locate  sources  of  intentional  and  unintentional  radiated  electromagnetic  energy  for 
the  purpose  of  immediate  threat  recognition.  Representative  examples  include,  but  are 
not  limited  to:  detection,  direction  findiip  (DF),  precision  emitter  location,  signal 
parametric  analysis,  threat  system  classifrcation. 

1 .3.2  Command  and  Control  Warfare  (C2W).  The  integrated  use  of 
OPSEC,  military  deception,  PSYOP,  EW,  and  physical  destruction,  mutually 
supported  by  intelligence,  to  deny  information  to,  influence,  degrade,  or  destroy 
adversary  C2  capabilities,  while  protecting  friendly  C2  capabilities  against  such 
actions. 


1.3.3  DoD  Annual  EW  Plan.  Tlw  annual  DoD  EW  Plan  compiles  EW, 
tactical  cryptologic,  and  tactical  SIGINT  technology  (multi-year)  RDT&E  and 
production  plans  in  the  President's  annual  budget  submitted  to  the  Congress.  The  plan 
is  prepared  by  the  Joint  Command  and  Control  Warfare  Center  (JC2WC),  the  Services, 
the  Defense  Intelligence  Ageiwy  (DIA),  National  Security  Agency  (NSA),  aiai  the  Joint 
Staff  (JS)  under  the  direction  of  the  Under  Secretary  of  l^fense  for  Acquisition  and 
Technology  {USD(A&T)]. 
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1.3.4  T&E  Oversight.  Programs  under  OSD  T&E  oversight  are  subject 
to  the  provisions  of  the  DoO  SOOO  series  which  require  that  each  program  have  a  Test 
aixi  Evaluation  Master  Plan  (TEMP)  and  that  OSD  approve  the  TEMP.  The  DoDI 
5000.2  requires  that  the  DOT&E  and  the  DT&E  jointly  publish  an  annual  listing  of 
major  and  other  designated  defense  acquisition  programs  for  OSD  T&E  oversight.  EW 
systems  under  OSD  T&E  oversight  include  all  ACAT I  programs  as  well  as  selected 
ACAT  n  and  m  programs. 

Aiq}endix  B  contains  the  definidon  of  other  significant  terms  used  in  this 
docun^nt. 
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2.  THE  DOD  TEST  AND  EVALUATION  PROCESS  FOR  EW  SYSTEMS 


2.1  Ovorriew.  A  DoD  Test  and  Evaluation  (T&E)  Process  for  Electronic  Warfare 
Systems,  shown  in  Figure  1.  is  a  five  step  iterative  process  that  provides  answers  to 
critical  EW  T&E  questions  for  decision  makers  to  support  decisions  for  each  phase  of 
the  acquisition  process. 


TEST  &  EVALUATION  PROCESS 
ns  COORDINATIQM  FUNCTION  TSE  flWPLEMEHTATlQH. 


(EXAMPLE  ACTIVITIES) 

•  REQUIREMENTS 

DEFINITION 

•  DIGITAL  MOOEUNG 

•  DESIGN 

•  PROTOTYPING 

•  SPECIFICATION 

•  REUABIUTY 

GROWTH 

•  PRODUCTION 

•  SUPPORT  EQMT 

•  TRAINING  EQMT 
«  OPERATIONAL 

CONCEPT 

•  LOGISTICS  8UPPT 

CONCEPT 


IDENTIFY  INFO 
NEEDED  BY 
DECISION  MAKER 
FROM  T&E 


PRE-TEST  ANALYSIS: 

'•  DEVELOP  ANALYTICAL 
TOOLS 

’  ESTIMATE  RESULTS 
•  0ETERMNETYPESK2NTYS 
OF  DATA  NEEDED 


eVAUMTION  — 


STEPS 

DECISION  MAKER 
WEIGHS  T&E 
INFO  AGAINST 
OTHER  PROGRAM 
INFORMATION 


iioeosmt 


STEP* 


lo■JecTl^p| 


(EXAMPLE  ACTIVITIES) 

•  TEST  ACTIVITY  PLANNING 

•  TEST  CONDUCT 

•  TEST  EVENTSAIATA  SOURCES: 
••  HISTORICAL 

..  COMPONENT  MEASUREMENT 
..  INTEGRATION 
..  HARDWARE*N-THE4.00P 
••  INSTALLED  SYSTEM 
••  FIELD  OR  OPEN  AIR 
••  SIMULATION/STIMULATION 

•  DATA  MANAGEMENT: 

••  COLLECTION 

..  REDUCTION 
••  ANALYSIS 
••  AUTHENTICATION 
••  DISTRIBUTION 

•  TEST  ACTIVITY  REPORTING 
••  DEFICIENCY  REPORTING 


•  POST  TEST  SYNTHESISE  EVALUATION 
•*  COMPARE  EXPECTEOMEASURED  OUTCOMES 
••  APPLY  TECHNICAL  AND  OPERATIONAL  JUDGMENT 
..  PREPARE  TECHNICAL  AND  OPERATIONAL 
ASSESSMENTS 


MOOCLS  /  SlUUlATiaN  SYSTEMS  USES  THROUGHOUT 


Figure  1.  The  Five  Step  T&E  Process  fcr  EW  Systems 


The  Process  is  contained  within  two  major  functions,  the  T&E  Coordination  Function 
and  the  T&E  Implementation  Function.  The  Process  can  involve  extensive  use  of 
modeling  and  simulation  systems.  It  provides  a  continuous  approach  to  the  problem  of 
managing  EW  system  acquisition  during  T&E.  Detailed  feedback  is  essential 
throughout  the  process  for  successful  EW  system  acquisition.  The  Process  establishes  a 
comprehensive  approach  to  T&E  that  may  be  applied  one  Oi  more  times  within  each  of 
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the  five  phases  of  the  DoD 
acquisition  process  shown  in 
Figure  2.  The  process  is 
structured  so  that  it  can  be 
used  at  any  tune  by  any 
program.  Its  application  and 
execution  are  based  iqx>n  EW 
T&E  information  needed  by 
decision  makers. 


2.2  Geno^.  The  Milestone  0 
(MS  0)  decision  marics  the 
beginning  of  the  acquisition  process  with  the  approval  of  the  Mission  Neoi  Statement 
(MNS).  The  EW  MNS  defines  projected  needs  for  an  EW  capability  in  broad 
operational  terms  of  mission  objectives  and  general  capabilities  providing  a  clear 
military  worth.  The  MS  0  decision  also  identifies  the  alternative  EW  concepts  or 
designs  to  be  smdied;  approves  the  start  of  Phase  0  -  Concept  Exploration  and 
Definition,  establishes  Phase  0  exit  criteria  to  be  evaluated  at  Milestone  I,  and  activates 
the  DoD  T&E  Process.  Milestone  decisions  are  documented  in  the  Acquisition 
Decision  Memorandum  (ADM)  for  Acquisition  Category  (ACAT)  I  EW  programs. 

2.2. 1  During  each  phase  and  at  each  milestone,  decision  makers  need 
information  from  several  sources  to  determine  the  best  course  of  action.  The 
information  needed  from  T&E  can  be  summarized  as  a  series  of  questions  to  be 
answered  during  the  process  in  each  phase.  The  questions  are  the  basis  for  the  T&E 
process  in  that  pb"se;  and,  in  this  document,  they  are  stated  in  the  context  of  the 
Process  as  they  relate  to  each  phase.  The  outcome  of  the  EW  T&E  process  is  technical 
and  operational  assessments  delivered  to  the  decision  maker  as  answers  to  those 
questions. 

2.3  T&E  Coordination  Function.  The  T&E  process  for  EW  systems  begins  with 
the  T&E  Coordination  Function,  which  has  three  parts;  the  Research,  Development  and 
Acquisition  (RDA)  Function  and  Steps  One  and  Five  of  the  Process. 

2.3.1  Research.  Development  and  Acquisition  (RDA)  Function  activities  are 
the  basis  for  T&E  activities  in  that  the  MNS  and  Operational  Requirements  Document 
(ORD)  contain  the  requirements  against  which  the  EW  system  must  be  evaluated  (see 
examples  in  Figure  3).  Although  requirements  generation  and  maintenance  are  not  part 
of  the  T&E  process,  the  T&E  community  should  be  involved  in  the  generation  of  these 
requirements  to  ensure  that  identified  requirements  are  testable,  measurable  and  can  be 
evaluated.  Other  RDA  activities  that  affect  T&E  range  from  specification  identification 
through  design  and  development  and  production  to  training  and  support  needs.  The 
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degree  to  which  each  of  tl^e  has  been 
demonstrated  must  be  considered  by  decision 
makers.  Answers  with  regard  to  the  achieven^nt 
of  EW  system  performance  objectives  come  from 
T&E.  Answers  to  other  questions,  such  as 
urgency,  military  effectiveness,  costs  and 
schedules,  come  from  other  sources  such  as 
Intelligence,  Program  Analysis  and  Evaluation 
(PA&E),  Legal,  or  Coiiq)troller. 

2.3. 1.1  Successful  EW  T&E  is  driven 
by  the  operational  environment  and  the  military 
mission  that  must  be  accomplished.  Hie  dominant 
measurement  of  the  operational  utility  is  how  well 
the  EW  system  is  able  to  perform  its  operational 
mission  over  time.  At  MS  0,  the  user  presents  the 
MNS  to  the  Defense  Acquisition  Board  (DAB).  If 
the  MNS  receives  favorable  consideration,  the 
DAB  authorizes  entry  into  Phase  0,  Concept 
Exploration  and  Definition,  from  which  are 
formulated  the  operational  suitability  and 
effectiveness  parameters  and  critical  technical 
parameters  (CTPs)  against  which  the  EW  system  is  tested  and  evaluated.  The 
objectives  of  Phase  0  are  to  explore  various  materiel  alternatives  to  satisfying  the 
documented  mission  need,  define  the  most  promising  system  concept(s),  develop  risk 
analyses,  and  develop  a  proposed  acquisition  strategy  and  initial  program  objective  for 
cost,  schedule  and  performance  for  the  most  promising  system  concept(s).  A  Cost  and 
Operational  Effectiveness  Analysis  (COEA)  and  ORD  are  prepared. 

As  the  acquisition  process  proceeds,  the  concepts  to  be  studied  in  Phase  0 
are  defined  and  the  engineering  and  production  prototypes  to  be  evaluated  in  Phases  I 
and  n  are  built.  In  Phases  m  and  IV  the  EW  system,  aixl  its  modifications  if 
appropriate,  must  be  evaluated. 


(EXAMPLE  ACTIVITIES) 

•  REQUIREMENTS 

DEFINITION 

•  DIGITAL  MODELING 

•  DESIGN 

•  PROTOTYPING 

•  SPECIFICATION 

•  RELIABILITY 

GROWTH 

•  PRODUCTION 

•  SUPPORT  EQMT 

•  TRAINING  EQMT 

•  OPERATIONAL 

CONCEPT 

•  LOGISTICS  SUPPT 

CONCEPT 


Figure  3.  Example  RDA 
Activities 


2.3. 1.2  The  RDA  Function  activities  list  in  Figure  3  includes  the 


following: 


2.3. 1.2.1  Requirements  Definition.  System  requirements  evolve  from 
i  :  MNS  through  the  COEA  and  the  ORD  to  the  Acquisition  Program  Baseline  (APB) 
and  the  Test  and  Evaluation  Master  Plan  (TEMP).  Requirements  must  be  clear, 
complete,  consistent,  feasible,  and  evaluateable.  They  are  derived  from  military  needs 
and  stated  as  technical  and/or  operational  suitability  and  effectiveness  parameters,  e.g.. 


6 


antenna  characteristics,  component  sensitivity,  operating  spectrum,  agility,  power 
levels.  The  requirements  are  articulated  as  objectives  and  thresholds,  e.g.,  minimiim 
required  jammer  effective  radiated  power  (ERP),  receiver/transmitter  bandwidths, 
maximum  dynamic  range.  They  must  be  neither  wholly  qualitative,  which  allows 
uncontrolled  personal  opinion  to  enter  the  process,  nor  wholly  quantitative,  forcing 
"failure"  of  a  system  thk  is  "good  enough”  but  not  perfect,  ^^tary  needs  (in  the 
MNS)  and  operational  requirements  (in  the  ORD)  are  the  criteria  u^  to  determine  the 
military  worth  of  a  system  relative  to  the  military  needs. 

(^rational  requirements  are  a  basis  for  CTPs,  used  by  the 
developmental  T&E  (DT&E)  program  as  the  basis  for  its  tests.  Operational 
requirements  are  also  a  basis  for  Measures  of  Effectiveness  (MOEs),  (e.g.,  missile 
miss  distances,  kill  probabilities,  detection  and  tracking  ranges),  used  by  the 
operational  T&E  (OT&E)  agency  as  the  basis  for  its  determination  of  operational 
effectiveness  and  suitability.  Measures  of  Performance  (MOPs),  derivnl  from  MOEs 
may  also  be  used  to  support  the  determination  of  operational  effectiveness  and 
suitability.  MOEs  and  MOPs  form  the  basis  for  the  operational  evaluation  which,  in 
turn,  shapes  the  operational  tests. 

2.3. 1.2.2  Digital  Modeling.  Digital  models,  implemented  on 
computers,  are  growing  in  importance,  use,  and  credibility.  A  digital  system  model 
(DSM)  is  a  computer  model,  or  software  equivalent,  of  a  system  under  development. 
Digital  models  used  in  RDA  activities  may  be  similar  or  identical  to  those  used  in  the 
T&E  Implementation  Function,  and  in  fact  may  be  first  created  during  Concept 
Exploration  and  Definition  and  updated  and  used  as  the  EW  system  proceeds  through 
development  and  T&E.  Models  ate  used  during  the  RDA  activity  to  evaluate  concept 
feasibility,  to  attempt  to  define  the  technical  limits  of  system  performance,  to  allocate 
requirements  and  functions,  to  plan  tests,  to  interpolate  test  results,  and  to  provide  a 
rigorous  evaluation  methodology.  These  models  may  be  single  thought  processes  or 
"back-of-the-envelope"  estimates,  or  they  may  be  sophisticated  simulations  of  system 
performance.  But  tlte  causal  relationship  between  military  effectiveness  and  system 
performance  must  always  be  developed,  validated  and  documented. 

2.3.1. 2.2.1  Those  digital  models  selected  for  use  in  the  EW  T&E 
process  must  be  appropriately  validated  and  certified.  Models  and  simulations  may 
range  in  scope  from  macroscopic,  i.e.,  operational  scenarios,  threat  engagements 
(many-on-many),  and  electromagnetic  interactions  among  systems,  to  a  high  degree  of 
fidelity  (microscopic),  i.e.,  detailed  technical  representation  of  electromagnetic 
propagation  and  signal  processing  functions  of  the  system  under  test  (SUT).  Any  or  all 
of  these  types  of  models  and  simulations  may  be  used  to  support  the  process  of 
translating  the  broad  operational  capabilities  as  described  in  the  MNS  into  system- 
specific  performance  requirements  that  may  be  demonstrated  through  T&E. 
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2.3. 1.2.3  Design.  System  Design  is  the  first  step  in  creating  the 
system.  EW  system  design  is  the  process  of  converting  specificatioos  for  an  EW 
capability  into  a  visual  or  mathematical  representation  of  the  system.  As  the  system 
clmracteristics  take  form,  it  is  veiy  importam  that  progress  on  the  system  design  is 
communicated  effectively  to  the  user  to  ensure  that  the  new  EW  o^bility  continues  to 
satisfy  the  operational  requirements.  System  design  is  perhaps  the  most  critical  step  in 
the  RDA  function  because,  once  built,  the  EW  system  is  much  more  difficult  to  modify 
to  correct  deficiencies  built  into  the  original  design.  Once  approved,  the  design  is 
converted  into  a  prototype  (working  model).  Testing  is  conducted  to  assess  whetlter 
the  EW  system  design  meets  the  requirements. 

2.3. 1.2.4  Prototyping.  Joint  Publication  1-02  states  that  a  prototype  is 
a  model  suitable  for  evaluation  of  design,  performance,  and  production  potential.  For 
the  purpose  of  this  process,  proto^pes  are  classified  as  engineering  and  production. 

2.3 . 1 .2.4. 1  Engineering  Prototype.  An  engineering  prototype 
(EP)  is  a  development  model  of  a  unit  that  is  close  to  production.  This  term  may  apply 
to  circuitry,  a  device  (black  box),  or  a  system,  and  may  be  in  a  breadboard  (technic^) 
configuration.  EPs  are  normally  used  in  Phase  I. 

2.3.1. 2.4.2  Production  Prototypes  fPPV  A  production  prototype 
is  a  final  model  of  a  design  before  the  pilot  unit  is  approved  for  production.  It  should 
be  highly  representative  of  final  equipment,  except  that  the  exact  manufacturing 
assembly  process  and  production  design  changes  may  not  yet  be  used  or  incorporated. 

It  is  suitable  for  conq)lete  evaluation  of  its  electrical  and/or  mechanical  form  and  may 
be  in  a  brassboard  (technical  and  operational  configuration.  PPs  are  normally  used  in 
Phase  n. 


2.3. 1.2.5  Specifications  are  the  values  that  convert  requirements  into 
design  terms.  They  must  clearly  and  accurately  relate  back  to  the  technical 
requirements  of  the  system.  An  audit  trail  fiom  the  requirements  documents  to  the 
specifications  ensures  that  the  system,  when  built  to  the  specification,  performs  as 
intended. 


2.3. 1.2.6  Reliability  Growth  is  a  process  in  which  the  reliability  of  the 
operational  system  improves  through  identification  and  correction  of  systemic 
reliability  failures.  During  Engineering  &  Manufacmring  Development  (EMD), 
laboratory  tests  on  prototypes  are  performed  to  uncover  component  reliability  failures, 
make  design  improvements,  and  project  a  reliability  point  estimate.  Later,  the  system  is 
deployed  and  field  data  are  collected.  The  validity  of  this  process  lies  in  the  activities 
associated  with  operational  failures.  If  the  user  is  aggressive  in  reporting,  cataloging 
and  investigating  reliability  failures,  and  commits  the  time  and  resources  to  their 
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correction,  reliability  will,  in  aU  lOcelihood,  improve.  If  these  steps  are  ixM  taken, 
reliability  cammt  inq)rove  and  will  probably  decrease  as  the  system  is  subjected  to 
more  maintenance  than  was  originally  intended. 

2.3. 1.2.7  Productjon  is  the  manufacturing  of  die  system  after  it  has 
been  funded,  fcnind  to  be  producible,  and  deemed  operationally  suitable  and  effective. 
The  capability  and  integrity  of  the  manu&cturing  process  to  proditee  systems  meeting 
the  system  d^ign  requirements  is  evaluated  and  measured  through  inspections  and 
testing.  The  attrilnites  of  a  producible  EW  system  design  are  that  it  can  be 
manu&ctured  economically  and  with  consistent  quality.  The  system  design  shouM 
address  the  manufteturing  facility's  variability  in  nutterial,  process,  and  personnel 
sources  and  require  that  the  man^acturing  processes  be  controlled  to  the  level 
addressed  in  tte  design. 

2.3. 1.2.8  Support  Equipment  is  that  category  of  ancillary  "things" 
necessary  to  sustain  the  EW  SIH'  or,  during  T&E,  to  support,  monitor  and  record  the 
test.  Support  equipment  includes  auxiliary  power  carts,  signal  generators, 
oscilloscopes,  video  and  still  cameras,  video  and  audio  recorders,  stripchart  recorders, 
and  numerous  other  items  without  which  tlte  system  could  not  operate  properly  or  be 
adequately  tested.  Support  equipuKut  requirentents  for  T&£  must  be  identified  in  Part 
V  of  the  TEMP  early  in  the  T^  process  to  ensure  availability  when  needed. 

2.3. 1.2.9  Training  Equipment.  Since  most  new  or  highly  modified 
systems  are  somewhat  unique,  the  personnel  who  use  and  maintain  the  system  must  be 
retrained  to  use  it  properly  and  safely.  Training  equipment  includes  items  such  as 
computer  workstations,  display  devices/consoles  for  troubleshooting,  and  test  bench 
mockups  of  the  system  that  either  permit  that  training  or  are  used  during  training  to 
instruct  the  operators  and  maintainers  in  the  correct  way  to  operate  and  maintain  the 
system. 


2.3.1.2.10  Operational  Concept.  The  operational  concept,  as  defined 
in  the  ORD,  is  the  planned  methodology  by  which  the  EW  system  is  to  be  used  and 
supported  during  peace  and  conflict.  The  ORD  provides  performance  parameters  in 
terms  of  operational  suitability  and  effectivemss  criteria  thresholds  and  objectives. 
Understanding  the  operational  concept  is  critical  for  both  DT&E  and  OT&E  personnel. 

2.3.1.2.11  Logistics  Support  Concept  is  the  plan  by  which  the  EW 
system  is  maintained.  It  includes  spares,  maintenance,  transportation,  and  support 
personnel  requirements.  Developing  integrated  logistics  support  requirements 
consistent  with  readiness  objectives,  system  design  and  resources  should  be  considered 
early  in  the  acquisition  process. 
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2.3.2  ?>  I'iiP  ONE  is  the  identification  of  T&E  information  required  by  the 
Milestone  decision  maker  (See  Figure  4).  The  required  information  consists  of 
performance  and  effectiveness  evaluations  of  how  well  the  system  meets  the  user's 
needs.  This  information  progresses  from  answers  concerning  proposed  alternative  EW 
concepts  in  Huise  0  to  answers  on  system  technical  performance  and  t^ierational 
suitabiliQr  and 
effectiveness  during 
Phase  n  and  beyond. 


2.3.2. 1  The 
required  information 
usually  centers  on  the 
current  SUT  which  is  in 
the  form  of  concepts, 
prototypes,  both 
engineering  and 
production,  and/or  the 
system  itself  depending 
on  the  acquisition  phase. 

Step  One  of  the  T&E 
process  is  initiated  by 
the  preceding  Milestone 
decision  of  the 
Acquisition  Process. 

2.3. 2.2  Areas  in  which  questions  can  be  expected  from  the  Milestone 
decision  maker  in  each  phase  are:  the  relevance  of  historical  data,  accuracy  and 
connectivity  of  stated  requirements,  adequacy  of  T&E  infrastructure  and  technology 
base,  testing  alternatives,  system  performamre  versus  validated  threats,  projected 
impact  of  the  system  on  battle  outcomes,  capabilities,  limitations,  feasibility,  preferred 
system,  and  Ti£  exit  criteria.  Association  of  each  question  with  an  acquisition  phase 
is  discussed  later  in  the  application  of  the  T&E  Process  to  the  Acquisition  Process.  The 
principal  outcome  of  this  Step  is  the  determination  of  evaluation  objectives. 

2.3.2.2.1  To  ensure  that  a  sound,  cost-effective  approach  is  implemented, 
the  T&E  resources  need  to  be  identified  as  early  in  the  process  as  possible-preferably 
prior  to  Step  One.  As  testing  proceeds  and  the  system  matures,  the  level  of  fidelity 
(accuracy)  of  required  data,  of  threats  and  surrogates,  and  of  the  test  environment 
increases  and  costs  escalate  rapidly.  Accordingly,  the  test  manager  should  carefully 
weigh  the  numbers  and  types  of  tests,  in  a  cost/benefits  analysis,  and  assess  required 
data  that  could  be  acquired  through  comparatively  lower  cost  modeling  and  simulation 
versus  data  that  must  be  acquired  through  the  more  costly  field  or  open-air  test  events. 


Figure  4.  Stq)  One:  Identification  of  Information 
Needed  by  Decision  Maker 
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2.3.2.3  A  Requirements  rnrrelation  Matrix  (RCM^  (similar  to  the  example 
illustrated  in  Figure  5)  can  be  used  to  provide  an  audit  trail  of  the  £W  jystem  under 


Figure  5.  Sample  Missile  Warning  System  Requirements 
Correlation  Matrix  (RCM) 

development.  Such  a  matrix  could  include,  but  not  be  limited  to,  a  comparison  of 
operational  requirements,  operational  performance  parameters,  and  key  system 
performance  parameters.  Models  and  simulations  can  be  used  to  assist  in  establishing 
objective  and  threshold  values  which  can  be  displayed  in  an  RCM.  The  sample  RC!M 
identifies  operational  requirements,  operational  performance  parameters  and  key  system 
performance  parameters.  These  characteristics  and  requirements  serve  as  the  foundation 
for  development  of  a  System  Maturity  Matrix  (SMM). 


2.3.2.4  A  System  Mamritv  Matrix  (SMM^  (similar  to  the  example 
illustrated  in  Figure  6)  is  an  acquisition  management  tool  that  can  be  used  to  highlight 
differences  between  the  required  objective/threshold  values  and  the  demonstrated 
values  resulting  from  scheduled  testing.  The  sample  SMM  contains  key  system  and 
technical  performance  parameter  thresholds  as  appropriate,  as  well  as  objectives,  at 
specific  points  in  time  within  the  development  process  of  the  system.  These  specific 
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thresholds  are  called  "exit  criteria”.  The  SMM  displays  these  parameters  necessary  to 
measure  progress  toward  meeting  the  operational  requirements.  At  the  milestone 
review  points^  decisknc  are  made  as  to  the  abili^  of  tiie  system  to  proceed  to  the  next 
phase  in  the  acquisition  process.  Exit  criteria  for  critical  parameters  are  requirements 
so  important  to  the  need  that,  if  not  satisfied,  will  result  in  cancellation  or  reassessment 
of  the  program.  The  characteristics  listed  in  the  SMM  will  likely  be  few  in  number. 
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Figure  6.  Sample  Missile  Warning  System  Mabirity  Matrix  (SMM) 

As  the  EW  system  progresses/matures,  sub-elements  or  new  characteristics  may  be 
added  to  these  matrices.  A  critical  characteristic  is  a  requirement  so  important  to  the 
need  that,  if  not  achieved,  will  result  in  cancellation  or  reassessment  of  a  program.  A 
threshold  is,  therefore,  the  value  a  critical  characteristic  must  meet.  It  becomes  an  exit 
criterion  when  coupled  with  a  specific  point  in  time  (a  particular  milestone  or  decision 
point)  when  it  must  be  attained.  Objectives  are  requirements,  although  not  critical,  that 
represent  user  desires  or  potential  improvement  above  thresholds.  After  T&E 
requirements  have  been  identified  and  are  considered  measurable,  pre-test  analysis  is 
performed  before  actual  testing  begins. 
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2.3.3  STEP  HYE  (See 
Figure  7)  is  where  the  decision 
maker  weighs  the  T&E 
information  against  other 
programmatic  information  to 
decide  a  proper  course  of  action. 

The  decision  will  bu  based  on 
criteria  of  militaiy  wtmh,  as 
well  as  conskteration  of  cost, 
funding,  urgency,  etc.  When 
associated  widi  an  acquisition 
milestone,  this  decision  is 
announced  in  an  Acquisition 
Decision  Memoranda  (ADM) 
which  outlines  the  future  course 
of  action  for  the  program  and  the 
SUT.  Additionally,  each 
milestone  decision  contains  the 
"exit  criteria"  for  the  next  Phase/Milestone. 

2.3.4  Feedback.  It  is  possible  that  the  decision  maker  asked  the  wrong  question 
or  that  the  questions  were  misunderstood  by  the  T&E  community.  The  decision  maker 
should  compare  the  information  contaii^  in  the  assessments  with  the  questions 
previously  asked  to  ensure  that  the  responses  are  adequate.  Problems  should  be 
highlight^  and  resolved.  The  decision  maker  can  then  rephrase  succeeding  questions 
to  ensure  better  understanding  of  the  information  needed. 

2.4  T&E  Implonentation  Function.  The  test  and  evaluation  implementation 
function  encompasses  the  three  steps  necessary  to  develop  the  information  needed  to 
prepare  the  assessments  used  by  decision  makers  in  Step  Five. 

2.4.1  Si  1  HP  TWO  (See  Figure  8)  is  the  pre-test  analysis  of  the  evaluation 
objectives  from  Step  One  to  determine  the  types  and  quantities  of  data  iweded,  the 
results  expected  or  anticipated  from  the  tests,  and  the  analytical  tools  need^  to  conduct 
the  tests  and  evaluations. 

2.4. 1 . 1  Pre-test  analysis  develops  the  analytical  tools,  allocates  test 
parameters  to  requirements,  estimates  test  results,  determines  the  types  and  quantities 
of  data  needed,  and  identifies  the  major  test  objectives. 
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Figure  7.  Step  Five:  Decision  Maker  Weighs 
the  T&E  Information  Against  Other  Program 
Information 
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2.4.1.2  The  use  of 
validated  nxxlels  and 
simulaticni  systems  during  pre¬ 
test  analysis  can  aid  in 
determi^ng:  1)  how  to  design 
test  scenario<s),  2)  how  to  set- 
iq)  the  test  environment,  3)  how 
to  properly  instrument  die  test, 
4)  how  to  man  and  control  the 
test  restnirces,  5)  how  best  to 
sequence  the  test  trials,  and  6) 
how  to  estimate  outcomes.  In 
this  stq>,  models  and 
simulations  are  used  to  estimate 
test  results. 


iMgure  8.  Step  Two:  Pre-Test  Analysis 

2.4. 1.3  The  end 

product  of  this  step  is  the  expected  outcome  of  the  system  under  test.  When  a 
determination  is  made  that  additional  data  are  necessary  and  major  test  objectives  are 
identified,  the  process  moves  to  Step  Three. 


2.4.2  .<rrKP  THREE. 

test  activity  and  data 
management,  (See  Figure 
9)  is  the  actual  test  activity 
planning,  test  conduct,  and 
data  management  practices. 
Given  the  data  requirements 
from  Step  Two,  T&E 
managers  determine  what 
valid  data  exist  in  historical 
files  that  can  be  applied  to 
the  SUT  and  what  new  data 
must  be  developed  from 
test  events.  They  plan  and 
execute  the  tests  necessary 
to  develop  the  data.  The 
historical  and  developed 
data  are  reviewed  for 
completeness  and  accuracy, 
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Figure  9.  Step  Three:  Test  Activity  and  Data 
Management 


authenticated,  and  forwarded  to  Step  Four  for  assessment  as  measured  outcomes. 
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2.4.2.1  Test  ac^vity  planniog  includes  gathering  the  required  test  articles 
and  test  support  equqnnent,  scheduling  facilities  and  identifyii^  the  climatic,  mission 
and  threat  environments. 

2.4.2.2  Test  conduct  uses  the  EW  T&E  capabilities  infrastructure  and  is 
the  culmination  of  test  activiQr  planning  and,  widi  the  exception  of  historical  data,  is 
the  execution  of  the  actual  test  events.  As  shown  in  Hgure  9,  test  activities  rai^e  from 
historical  searches  of  the  performance  of  like  or  similar  components,  subsystems  and 
systems  m  actual  hardware  tests  of  conq)oneiit  incoming  parts  through  multi-system, 
open-air,  qperationaily  realistic  ”free-play”  scenarios.  Data  gathered  during  the  test,  or 
from  historical  searches,  are  input  to  the  data  management  activity  for  processing. 

The  Reliance  Study  defines  EW  T&E  capabilities  as  those  '‘esources  used 
for  the  evaluation  of  electronic  countermeasures  (ECM);  electronic  counter- 
countermeasures  (ECCM):  suppression  of  enemy  air  defense  (SEAD);  and  command, 
control,  communications,  and  intelligence  (C3I)  components,  subsystems,  systems,  and 
EW  functions  of  federated  or  integrated  avionic/vetronic  suites.  EW  testing  includes 
operational  test  considerations  for  missions  and  warfighting  requirements  of  the  Army, 
Navy,  Air  Force  and  Marine  Corps  and  involves  evaluation  of  their  vehicles  and  EW 
equipment  for  required  performance,  mutual  interference,  and  detectability  by  the 
enemy. 


2.4.2.2.1  Historical  Test  Data.  The  initial  step  in  any  EW  test  activity 
stu)uld  be  the  examination  of  previous  test  data  (or  descriptive  summaries)  stored  in 
historical  archives  to  ascertain  the  utility  of  previous  test  results,  i.e.,  does  the  required 
new  EW  capability  share  sufficient  commonality  with  any  predecessor  system  such  that 
some  testing  can  be  diviated.  Component,  subsystem  aixi  system  data  from  like  or 
similar  components,  subsystems  or  systems  must  be  examined  first  for  applicability, 
and,  if  applicable,  to  reduce  the  amount  and/or  type  of  hardware/software  testing 
necessary  for  evaluation.  Further,  modeling  and  simulation  of  historical  test  data,  in 
lieu  of  actual  testing  of  the  currem  SUT,  should  be  performed  to  fill  historical 
information  voids.  This  is  a  highly  cost  effective  procedure  that  leads  to  the 
determination  of  what  data  are  lacking  and  needed  from  new  test  and  evaluation  events. 

2.4.2.2.2  Component  Measurement  Test  Events  often  involve  the  use 
of  specialized  capabilities  to  explore  and  evaluate  advanced  technologies,  e.g.. 
Microwave  Monolithic  Integrated  Circuits  (MMIC),  Very-high-speed  ICs  (VHSIC), 
Microwave  Power  Modules  (MPM),  Digital  RF  Memory  devices  (DRFM),  Acoustic 
Charge  Transport  (ACT),  and  are  usually  the  first  test  events  performed  during  the 
development  and/or  buildup  of  the  system.  Examples  include  incoming  parts 
inspection,  thermal,  acoustic  and  vibration  cycling,  power  requirement,  and  heat 
generation  tests.  For  EW  systems,  measurement  resources  provide  those  specialized 
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Cj4>abilities  cited  above.  They  generally  fall  into  the  sub-cat^ories  of  antenna 
measurement*  RCS  measurement,  infrared  (IR)/laser  signanire  measurement,  and 
electromagnetic  inter£erence/conq>atibiUty  (EMI/EMC)  test  cq)abilities. 

2.4.2.2.3  Integration  Test  Evente  test  £W  conqwnents,  subsystems  and 
systems  combined  with  other  elements.  The  other  elements  may  be  other  parts  of  the 
same  system  or  different  systems  altogether  widi  which  the  SUT  must  operate.  These 
tests  are  used  to  evaluate  individual  hardware  and  software  interactions  with  each  other 
and  with  the  entire  avionic/vetronic  suite  and  are  frequently  conducted  in  integration 
laboratories  speci&ally  designed  to  test  the  SUT  intqsiated  with  odier  systems  or 
functions.  Integration  laboratories  are  generally  weapon  system  specific  (excqit  for 
associat«l  environment  generators)  and  are  us^  from  the  beginning  of  an  EW  system’s 
development  through  avionics/vetronics  integration  and  fielding.  These  laboratories 
often  employ  a  varieQr  of  digital  models,  simulations,  and  stimulations  to  generate 
scenarios  and  electromagnetic  backgrounds  at  or  near  real  time. 

2.4.2.2.4  Hardware-in-the-loop  (HTTL)  Events.  HTTL  tests  use 
elements  of  the  SUT  in  combination  with  software  to  examine  the  performance  of  those 
elements  before  the  entire  system  is  available  or  >^n  a  specific  capability  cannot  be 
tested.  For  EW  systems,  these  events  are  conducted  indoors  in  a  secure  environment  to 
test  the  systems  against  manned,  closed-loop,  and  open-loop  threat  simulators.  These 
tests  provide  unique  opportunities  to  evaluate  EW  systems  hardware  at  different  stages 
of  developmem  (e.g.,  breadboard,  brassboard,  prototype,  or  production),  possibly 
years  before  the  host  platform  is  available,  thereby  facilitating  concurrent  developmem 
of  the  EW  systems  or  functions,  other  systems,  and  the  platforms/vehicles  themselves. 
HTTL  testing  also  allows  production  systems  to  be  tested  under  controlled  axxl 
repeatable  test  conditions,  thus  providing  an  inexpensive  complemem  to  flight  testing. 
Additionally,  the  controlled  environment  readily  lends  itself  to  EW  technique 
optimization  and  closed-loop  operational  effectiveness  evaluation. 

2.4.2.2.5  Installed  Systems  Test  Events  provide  capabilities  to 
evaluate  SUTs  and  functions  that  ate  installed  on  and  integrated  with  their  host 
platforms.  These  tests  can  occur  in  indoor  facilities  such  as  EW  anechoic  chambers,  in 
which  free-space  radiation  measurements  are  made  during  simultaneous  operation  of 
the  EW  system  and  other  host  platform  systems,  or  climatic  chambers  or  as  outdoor 
DT  and  OT  tests.  Chambers  provide  secure  sites  to  evaluate  the  capabilities  and 
limitations  of  the  system  against  simulated  ami  stimulated  inputs.  In  an  anechoic 
chamber,  for  example,  the  SUT  is  stimulated  by  threat  signal  generators  and  its 
responses  are  evaluated  to  provide  critical  information  regarding  integrated  system 
performance.  Climatic  chambers  permit  examination  of  SUT  capabilities  in  varied 
climates  without  having  to  transport  the  SUT  to  those  naturally  occiuring  climates. 
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2A.2.2.6  Field  or  Open  Air  Test  Events.  The  tenn  "field  test"  or  open 
air  test  refers  to  any  test  conducted  outdoors.  It  includes  surfoce  (land  and  sea), 
undersea,  airborne  and  sp2u:ebome  testing.  Field  tests  are  conducted  where  it  is 
feasible,  safe,  and  secure  to  test  all  or  part  of  the  SUT  in  an  enviromnent  that  is 
normally  more  realistic  than  any  attainable  indoors.  Held  tests  may  allow  the  SUT  to 
be  operated  more  closely  to  its  operational  conditions.  However,  particularly  with  EW 
systems,  field  tests  may  provide  less  insight  into  the  (purely  technical)  performance  of 
a  system  because  indoor  facilities  are  the  only  place  high  density,  high  fidelity  threat 
signals  can  be  generated. 


2.4.2.2.6.1  Tlte  Reliance  Study  has  divided  open  air  test  resources 
into  two  subcategories:  EW  test  ranges  and  airborne  testbeds.  1)  Open  air  EW  test 
ranges  are  highly  instrumented  facilities  with  high  fidelity  threat  simulators  and  real 
systems,  and  are  primarily  used  to  test  systems  and  functions  installed  either  in  a 
testbed  or  the  intended  host  aircraft/vehicie.  Open  air  testing  provides  the  most 
operationally  realistic  environment  in  which  to  evaluate  a  SUT.  Real  world  phenomena 
encountered  during  range  testing  include  terrain  effects,  multipath  propagation, 
electromagnetic  interference  (EMI)  effects,  etc.  2)  Airborne  testbeds  are  normally 
large  airfnunes  designed  for  spread-bench  installation  and  testing  of  EW  and  other 
avionics  systems.  They  permit  flight  testing  of  EW  techniques,  components, 
subsystems,  systems,  or  entire  avionics  suites  in  their  early  stages  of  development 
and/or  modification,  often  prior  to  the  availability  of  prototype  or  production  hardware. 

2.4.2.2.7  Simulation/Stimulation  Events.  Simulation  and  stimulation 
events  are  used  extensively  in  the  DoD  test  process.  They  can  be  applied  to  computer 
or  physical  working  models  or  the  SUT.  They  may  be  real  time  or  non-real  time 
models.  Effective  use  of  credible  models  and  their  sinnilation/stimulation  events 
provides  cost  effective  T&E. 

2.4.2.2.7.1  According  to  the  EW  T&£  Reliance  Study,  digital 
models  and  computer  simulations  represent  EW  systems,  host  platforms,  other  friendly 
players,  the  combat  environment,  and  threat  systems.  These  models  run  interactively  in 
real  or  simulated  time  and  space  domains,  along  with  other  factors  of  a  combat 
enviromnent.  Specific  computer  simulations  are  constructed  at  various  levels  of  detail, 
corresponding  to  the  level  of  technical  complexity  they  support  (i.e.,  engineering, 
platform,  mission,  or  campaign).  Presently,  there  are  numerous  computer  simulations 
in  use  (e.g.,  AASPEM,  SUPPRESSOR,  ALARM,  and  Enhanced  SAMS);  however, 
they  do  not  share  a  common  architecture.  For  example,  AASPEM,  the  Advanced  Air- 
to-Air  System  Performance  Evaluation  Model,  models  air-to-air  engagements, 
including  beyond-visual-range  maneuvering;  close-in-combat  air-to-air  tactics;  sensor 
detection  and  tracking;  missile  lock-on,  launch,  fly-out,  firing,  detonation,  and  kill; 
gun  firing;  laser  firing;  and  defensive  reaction  to  weapons  while  ALARM,  the 
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Advanced  Low  Altitude  Radar  Model,  is  a  radar-range-equation-based  detection  model 
that  includes  the  effects  of  ground  clutter,  terrain  masking,  multipath,  diffraction, 
atmospheric  attenuation  and  jamming  against  several  types  of  radar  systems,  e.g.,  MTI, 
pulse  Doppler,  and  CW.  While  both  of  these  exaiiq>les  describe  very  complex  models 
that  possess  a  high  degree  of  specificity,  their  purpose  and  use  in  the  T&E  process 
would  be  for  very  different  reasons  and  probably  at  different  phases  in  the  Acqubition 
process,  i.e.,  ALARM  would  appear  to  be  more  ^propriate  for  assessing  technical 
performance  while  AASPEM  would  likely  be  used  closer  to  OT&E  to  assess 
operational  performance. 

2.4.2.2.7.2  A  greater  capability  to  simulate  existing  and  planned 
military  systems  and  the  threats  they  are  desigW  to  counter  is  needed.  The  Joint 
Modeling  and  Simulation  System  (J-MASS)  program  is  addressing  these  problems  and 
should  be  contacted  for  use  of  their  standard  modeling  architecture  and  simulation 
support. 


2.4.2.3  Data  Management.  Data  recorded  during  test  events  are  often  not 
in  a  form  best  suited  for  analysis.  Several  steps  are  taken  to  make  the  data  more  usable: 

2.4.2.3.1  Data  Collection  and  Reduction.  Most  data  are  recorded 
"raw”  and  scaled  to  match  the  recording  capabilities  of  the  analog  or  digital  recording 
system  without  regard  to  the  actual  magnitude  of  the  data.  This  first  step  in  the  data 
management  process  is  assembling  data  from  all  sources  and  reducing  them  to 
engineering  values. 

2.4.2.3.2  Analysis  ami  authentication  ensures  that  all  data  accurately 
reflect  the  operation  of  the  SUT.  Data  from  multiple  sources  are  compared  for 
agreement,  data  dropouts  are  filled  in  where  possible  and  questionable  data  are 
compared  with  other  sources  for  reasonableness.  The  data  are  then  distributed  for 
further  use  in  the  synthesis  and  evaluation  step. 

2.4.2.4  Deficiency  Reporting  is  the  process  of  formally  documenting 
failures  to  meet  required  performance  thresholds  or  objectives,  human  factors 
limitations,  safety  concerns,  etc.  Deficiency  reports  are  forwarded  to  the  program 
office  for  correction.  Procedures  for  deficiency  reporting  differ  among  the  Services, 
but  because  of  the  critical  importance  of  deficiency  reporting  during  field  testing,  it 
should  in  all  cases  be  a  clearly  defined,  formal  process.  Evaluation  of  the  impact  of  the 
deficiency  on  suitability  and  effectiveness  must  be  part  of  the  deficiency  reporting 
process.  Merely  quoting  specifications  as  a  justification  for  submitting  a  deficiency 
report  may  lead  to  costly,  time-consuming  changes  that  are  not  operationally  required 
or  that  preclude  making  other,  more  valuable  changes.  For  example,  the  evaluators 
should  assess  how  the  deficiency  affects  operational  mission  accomplishment.  Does  the 
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deficiency  require  more  spares,  more  manpower,  or  longer  down  times  tban  can  be 
afforded  in  a  military  operation?  And  finally,  can  the  user  work  armind  these 
limitations  and  make  effective  use  of  the  system? 

2.4.2.S  The  end  product  of  this  step  is  measured  outcomes  in  the  form  of 
T&E  activity  reports  which  are  provided  to  the  analysts  for  Step  Four. 

2.4.3  STEP  FOUR,  post  test  synthesis  and  evaluation,  is  the  combination  of 
the  measured  outcomes  of  Stqp  Three  with  the  expected  outcomes  from  Step  Two, 
tempered  with  technical  and  operational  judgement.  The  output  of  Stq)  Four  is  the 
answers  to  the  questions  developed  at  Step  One.  See  Figure  10. 


2.4.3. 1 

When  measured 
outcomes  differ  from 
expected  outcomes,  the 
test  conditions  and 
procedures  must  be 
reexamined  to  determine 
if  the  deviations  are 
real,  that  is,  due  to 
unexpected  performance 
of  the  SUT,  or  are 
caused  by  test 
limitations  such  as  a 
lack  of  fidelity  in 
computer  simulation, 
non-availability  of 
support  assets,  or  less 
than  full  system 
availability.  If  the 

differences  are  due  to  test  limitations,  the  effect  of  the  limitations  must  be  evaluated,  if 
possible,  and  judgement  used  to  estimate  true  system  performance.  However,  siiK:e 
this  may  involve  extrapolation  of  test  data,  it  is  inherently  risky.  Despite  the  additional 
cost  and  time,  retesting  is  usually  prudent. 

In  this  step,  models  and  simulations  are  normally  used  to  process  test  data  and 
to  evaluate  system  performance  and  effectiveness  using  data  obtained  from  the  tests. 
The  assumptions  of  tactics,  environment,  system  performance,  and  support  must  be 
carefully  chosen  and  fully  described  and  documented. 


Figure  10.  Step  Four:  Post  Test  Synthesis  and 
Evaluation 
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2.4.4  Feedback.  As  with  the  decision  maker,  feedback  is  necessary  widiin  the 
T&E  process  to  ensure  the  quality  of  the  outpuL  But  unlike  the  decision  maker,  who 
needs  to  know  whether  the  questions  have  b^n  answered  satisfactorily,  here  the 
evaluators  must  ensure  that  the  data  are  sufficient  to  answer  the  questions  posed  by  (or 
as  understood  from)  the  decision  maker.  Their  satisfaction  with  the  test  report  data 
must  be  transmitted  as  feedback  to  the  persons  responsible  for  the  pre-test  analysis  to 
ensure  that  both  the  current  test  and  the  process  itself  are  as  complete,  effective  and 
efficient  as  possible. 

This  st^  concludes  with  the  preparation  of  technieai  and  operational 
assessments  which  answer  the  questions  from  the  decision  maker  at  Step  One. 

2.4.5  The  T&E  process  concludes  at  STEP  FIVE,  as  noted  above,  where  the 
decision  maker  weighs  the  T&E  information  against  other  program  informanon  and 
assesses  the  progress  of  the  EW  SUT. 


20 


3.  EW  SYSTEMS  T&E  DURING  PHASE  0  -  CONCEPT  EXPLORATION  AND 
DEFINITION  (CE&D) 


3.0.1  Milestone  Q-Concept  Studies  Approval.  The  Milestone  (MS)  0  decision 
begins  Phase  0  in  which  alternative  EW  concepts  are  to  be  explored  to  i<knt!fy  the 
most  promising  potential  solution(s)  to  validate  user  needs.  T&E  concerns  at  MS  0  ate 
data  diat  can  be  developed  during  Phase  0  that  su{^rt  a  MS  I  decision  with  respect  to 
the  concqpts  and  the  develofanent  of  recommended  T&E  "exit  criteria"  diat  are 
established  and  presented  in  die  MS  0  Acquisition  Decision  Menxnandum  (ADM). 


3.0.2 


Phase  O-Concent  E 


1  and  Defipition. 


3.0.2. 1  Phase  Description.  The  main  objective  of  the  CE&D  phase  is  to 
arrive  at:  1)  a  preferred  system  concept,  or  concepts,  through  study  and  comparison  of 
alternative  concepts  including  a  proposed  acquisition  strategy  and  program  objectives 
for  the  most  promising  system  concept(s);  2)  a  determination  of  system  performance 
and  design  characteristics;  and  3)  an  sqipraisal  of  the  operational  effectiveness  and 
suitability  of  each  EW  concept. 

3.0.2.2  l&E.  The  main  objective  of  T&E  in  this  phase  is  to  assist  in 
defining  and  selecting  a  preferred  EW  System  concept,  candidate  technologies,  and 
critical  operational  characteristics. 


3.0.2.2.1  Developmental  Test  and  Evaluation.  DT&E  comprises  the 
major  test  events  and  evaluation  actions  within  this  phase.  The  primary  purpose  of 
DT&E  at  this  point  is  to  determine  whether  the  proposed  EW  concepts  are  feasible. 
Alternative  concepts  are  analyzed  against  required  key  technical  and  operational 
performance  parameters,  e.g.,  is  nanosecond  response  time  required  or  will 
microsecond  response  time  suffice?  Test  resources  and  tools  required  to  suppon  test 
and  evaluation  of  any  new  EW  system  that  may  be  developed  are  identified.  At  this 
juncture  within  most  EW  programs,  because  there  is  usually  little  EW  hardware  to  test 
at  this  stage,  much  of  the  smdies,  analyses,  concept  comparisons,  design  work,  and 
performance  aj^raisals  are  done  through  the  use  of  modeling  and  simulation.  In  some 
cases,  however,  as  a  result  of  an  Advanced  Technology  Demonstration  (ATD) 
program,  a  brassboard  or  breadboard  may  be  available  for  testing.  Considerable  testing 
of  concepts  and  some  testing  of  subsystems  occurs  in  this  phase. 

3.0.2.2.2  Operational  Test  and  Evaluation.  Significant  OT&E  will  not 
likely  occur  during  this  phase.  The  emphasis  in  this  phase  is  in  proving  out  the 
particular  concept  and  developing  cost  effective  alternatives  that  meet  the  user’s  needs. 
To  that  end,  the  Operational  Test  Agency  (OTA)  reviews  program  documentation  and 
develops  a  working  relationship  with  the  user  in  order  to  understand  the  operational 
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lequiiements  and  define  the  critical  issues.  The  OTA  will,  as  a  minimum,  prepare  an 
initial  OT&E  outline  ami  participate  in  acquisition  planning  to  ensure  that  schedules 
and  long-lead  items  meet  OT&E  objectives.  The  OTA  may  be  required  to  prepare  an 
early  operational  assessnwnt  (EOA)  of  critical  systems,  st^systems,  and  components 
may  be  necessary  to  ascertain  that  risk  and  operating  -environment  are  within 
acceptable  limits. 

3.1  STEP  ONE.  Identify  Information  Needed  by  Decision  Makers  from  T&E. 
As  a  minimum,  T&E  addresses  the  (generic)  questions  as  deleted  in  Hgure  11  and, 
more  specifically  as  listed 
below  as  they  might  relate  to 
each  EW  concept  or  concept 
comparison: 

(1)  What  T&E  data 
exist,  e.g.,  are  there 
historical  test  data 
available  on  previous 
EW  concepts,  jammer 
systems,  ECM 
techniques,  etc.  that 
would  preclude 
"reinventing  the 
wheel"?  Does 
preliminary  analysis 
conclude  that  the 
concepts)  will  work? 

What  is  the 
confidence  level  in 

this  assessment?  Do  the  available  historical  data  on  the  testing  of  platform  self¬ 
protection,  for  exanq)le,  enhance  the  confidence  that  an  evolutionary,  rather 
than  revolutionary,  approach  is  preferred? 

(2)  Can  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  EW  T&E  infrastructure/technology  base  permit 
evaluation?  Approximately  what  is  the  cost  and  time  frame?  If  not, 
approximately  what  is  the  cost  and  time  frame  to  create  the  infrastruemre  and 
technology  base  to  adequately  assess  the  EW  concept(s)?  Does  the  existing 
technology  base  support  concurrent,  rather  than  sequential,  testing  of  parallel 
technologies? 
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Hgure  11.  Phase  0  T&E  C^estions 


22 


(4)  What  arc  coiicept(s)  capabilities  and  limitations  versus  threats,  e.g.  does  the 
.  ECM  concept  adequately  address  the  n»oqpulse  radar  direat? 

(5)  How  can  T&E  Ssvorably  inquict  risk  management  or  risk  rediKdon? 

(S)  What  alternatives  exist  to  testix^test  assets?  What  are  the  risks  associated 
widi  using  the  alternatives? 

(7)  Has  the  preliminary  TEMP  been  approved?  Are  the  critical  technical 
parameters,  die  mininium  accqitable  operational  requirements,  and  die  critical 
system  characteristics  included? 

<8)  Is  each  proposed  EW  concept  fei^ible?  Is  one  concept  preferred,  e.g.,  does 
an  onboard  EW  system  provide  more  or  less  survivability  of  the  host  platfonn 
than  an  integrated  onboaid/offboard  system? 

(9)  What  are  the  recommended  Phase  I  T&E  "exit  criteria"? 

(10)  If  an  EOA  is  initiated  by  the  OTA  at  this  stage,  does  it  address  resource 
availability  and  planning  for  projected  OT&E  test  events? 

3.2  STEP  TWO.  Pre-test  Analysis.  Pre-test  Analysis  in  EW  CE&D  places 
maximum  emphasis  on  the  development  of  expected  outcomes  after  determining  the 
major  types  of  tests  needed,  i.e.,  open-loop  or  closed  loop,  HTTL,  integration 
laboratory,  flying  testbed,  open  air  or  range.  Among  the  analytical  tools  developed 
during  pre-test  analysis  are  digital  models  and  simulations  of  elements  of  contending 
EW  concepts  that  are  sufficiently  representative  of  the  SUT  to  permit  conceptual  design 
trade-off  studies.  Conducting  pre-test  analysis  assists  in  structuring  test  conditions  prior 
to  the  conduct  of  test  events.  The  expected  response  of  the  SUT  can  be  used  to  design 
test  trials  to  evaluate  system  effectiveness  and  suitability.  For  the  initial  test  set-up,  test 
requirements  are  derived  from  the  system  developmem  and  engineering  activities. 

3.3  STEP  THREE.  Test  Activity  and  Data  Management.  This  step  emphasizes  a 
testing  approach  to  prove  out  the  feasibility  of  alternative  EW  concepts.  Test  planning 
involves  preparation  of  the  acquisition  program  baseline  (APB),  the  preliminary  TEMP 
and  an  initial  OT&E  outline.  The  actual  tests  should  be  strucnired  to  provide  data  for 
evaluation  of  acquisition  risk  and  decision  making.  Testing  in  the  CE&D  phase  is 
conducted  to  determine  whether  the  alternative  EW  concepts  can  meet  the  operational 
need,  e.g.,  can  the  missile  warning  system  meet,  or  exceed,  the  capability  to 
discriminate  among  at  least  the  top  10  (in  priority)  threat  missiles  against  which  it  is 
designed  to  operate.  Alternative  concepts  are  tested  against  the  requirements  outlined  in 
the  ORD  and  APB,  ami  listed  in  the  RCM  for  tracking.  Test  events  are  coiuiucted  at 
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facilities  commcmunte  widi  EW  T&E  capabilities  required  for  that  event.  In  the 
CE&D  Phase  the  following  may  be  considered. 

Histnrical  Teat  Data.  The  initial  step  in  any  EW  test  activity  should  be  the 
examination  of  previous  test  data  (or  descriptive  summaries)  stcned  in  historical 
archives  to  ascertain  the  utility  of  previmis  test  results,  i.e.,  does  the  retpiired  new  EW 
capability  share  sufficient  commonality  widi  ai^  predecessor  system  such  that  some 
testing  can  be  obviated.  Develcqnnent  plazming  agencies.  Service  analysis  ^encies  (Uke 
the  Air  Fotce  Center  Technology),  and  hardware  contractors  should  be  consulted 
Driien  researching  available  system  or  subsystem  histmkal  data  bases  or  data  bases  that 
should  share  at  military  laboratories.  The  use  of  TECNET,  INTERNET, 
Worldwide  WEB,  and  USENET  ate  sug^sttd  for  access  to  repositories  of  EW  data. 

A  thorough  search  of  available  test  data  sources  could  reduce  die  cost  of  die  T&E 
program  planned  for  the  potential  new  systenL  Further,  modeling  and  simulation  of 
historical  test  data,  in  lieu  of  actual  testing  of  the  current  SUT,  should  be  performed  to 
fill  historical  information  voids.  This  is  a  highly  cost  effective  procedure  that  leads  to 
the  determination  of  what  data  are  lacking  and  needed  from  new  test  and  evaluation 
events. 


Component  measurement  test  events  are  less  likely  to  occur  during  this 
phase  than  in  subsequent  phases.  However,  in  some  cases  actual  equqiment  in 
brassboard  or  proto^pe  configuradons  may  be  available  for  measurement  test  events. 

Tnteyration  testing  may  be  used  to  verify  fairly  complex  concepts  that 
require  sensor  fusion  (to  perform  a  more  accurate  Electronic  Support  function  such  as 
precision  emitter  location,  for  exanqile)  or  that  have  operator  di^lays  and  operator 
actions  that  need  to  be  evaluated  with  real  time,  man-in-the-lot^  simulations.  Emphasis 
should  be  on  lessons  learned  and  their  integration  with  data  available  from  computer 
simulations.  Test  efforts  should  make  efficient  use  of  conqmter  simulation  such  that 
sufficient  pre-test  analysis  can  be  accomplished  on  those  results  expected  from  the 
actual  laboratory  testily.  This  allows  ctmquuison  of  measured  results  with  expected 
outconus  to  validate  desired  test  conditions. 

Hardware-in-thft-T/OTp  testing  is  important  in  this  phase  in  determining  the 
feasibility  of  each  EW  concept  under  test  and  in  evaluating  its  military  worth.  That  is, 
what  is  the  value  added  by  the  implementation  of  the  EW  concept?  For  concepts  that 
have  matured  to  a  hardware  stage,  HITL  test  events  can  be  used  to  refine  designs, 
check  system  performance  in  operational  environments,  and  work  out  early  problems 
by  using  either  flexible,  generic  EW  simulatois/stimulators  or  commercial  laboratory 
equipment,  such  as  oscilloscopes,  synthesizers,  spectrum  analyzers,  signal  generators, 
etc. 
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Installed  System  Test  as*n  Event*  may  not  be  conducted  because  few 
concepts  will  have  reached  die  iimallation  sta^  by  this  Phase.  If  possibte,  however, 
1ST  events  are  used  to  look  for  optimum  solutions  to  stated  needs. 

Reld/Qpen  Air  Tests  may  or  may  not  be  conducted  during  this  phase 
depending  on  the  availability  of  prototype  iu^ware.  however,  advanced  technology 
development  (ATD)  programs  have  produced  brassboaxd  hardware,  it  may  be  installed 
in  tesAeds  to  conduct  fiefal  testing.  Data  may  be  obtained  in  this  manner  to  check 
equipment  toleration  and  help  verify  non-TOd-time  ctmqputer  siimilatitnis  from  the 
CE^  phase. 

Simiilarion/Stitnulation  Rvents.  Simulation  and  stimulation  events  are  used 
extensively  in  the  DoD  test  process.  They  can  be  applied  to  ctmqjuter  or  physical 
working  models  or  the  SUT.  They  may  be  real  time  or  non-real  time  nio^ls.  Effective 
use  of  credible  models  and  their  simulation/stimuiation  events  provides  cost  effective 
T&E.  Candidate  system  concepts  to  provide  the  needed  mission  capability  can  be 
analyzed  using  conqniier  simulations  of  mission  scenarios  integrated  with  models  of  the 
system  concepts.  The  use  of  ALARM  (described  in  paragraph  2.4.2.2.7.1),  or  a 
similar  model,  during  this  phase  should  be  considered.  ALARM  is  appropriate  for 
assessing  tarhnicai  characteristics. 

Data  managemmt  occurs  during  this  phase.  In  this  step,  the  data  on  each 
EW  concept  are  converted  from  raw  data  collected  by  the  testers  into  measured 
outcomes  which  are  analyzed  for  validity  and  then  authenticated. 

3.4  STEP  FOUR.  Post-test  Synthesis  mid  Evaluation  directly  identifies  the 
performance  capabilities  of  the  EW  system.  In  the  CE&D  phase,  the  concepts  identified 
in  the  COEA  are  judged  accordii^  to  tlwir  merit  in  meeting  military  needs  as  stated  in 
the  MNS  and  users'  requirements  as  stated  in  the  ORD.  Assessments  identify  tte 
MOPs  for  the  systems  as  well  as  any  shortcomings/deficiencies.  The  measured 
outcomes  of  the  tests  are  conqiared  to  the  eiqiected  outcomes  established  in  Step  Two. 
In  this  phase  the  greatest  disparity  between  expected  and  measured  outcomes  occurs. 
Measured  outcomes  of  testii^,  when  coupled  with  expected  results  hrom  computer 
simulation,  provide  the  added  benefit  of  refining  the  simulations  and  models  and 
updating  their  parameters  in  order  to  better  validate  the  model's  accuracy.  At  the 
coiK:lusion  of  the  post  test  synthesis  and  evaluation  step,  all  test  objectives  will  have 
been  analyzed  and  demonstrated  values  will  have  been  documented.  The  final  output  of 
this  step  must  be  T&E  information  for  the  decision  makers  concerning  the  performance 
parameters,  the  feasibility  of  each  alternative  £W  concept,  the  availability  of  EW  test 
resources,  and  the  adequacy  of  the  technology  base  to  support  the  desired  concept. 
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3.5  STEP  FIVE.  Oedsion  Maker  Wdglu  T&E  Infurmatloii  Against 
Pn^ram  InfMmatkm.  Tbc  ouqmt  of  this  step  is  an  infonned  decision  on  tbe  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 

3.5.1  Milestone  I  Decision  Point.  The  results  of  the  Concept  Exploration  and 
Definition  Iliase  are  reviewed  and  approved  by  die  Milestone  Decision  Authority 
(MDA).  During  this  review,  the  MDA  confinm  that  snidy  efforts  support  die  need  for 
a  new  program,  that  the  system  concqitfs)  shows  evidence  of  sadsfyhig  the 
requirements,  diat  the  threat  assessment  has  been  validated  as  requhed,  that  the 
proposed  EW  concqit  is  producible  and  affordable,  and  diat  adequate  resources, 
including  T&E  resources,  can  be  programmed  to  stqiport  an  acquisition  program. 

The  ADM  documents  the  decision  to  initiate  an  acquisition  program  and  enmr  into 
Phase  1  (DEMVAL).  It  also  tqiproves  the  Concept  Baseline  that  was  established 
through  the  analysis  of  the  alterative  EW  system  concepts.  Program>specific  exit 
criteria  that  must  be  met  during  the  DEMVAL  Phase  are  established. 

3.5.2  rinriifnentatinn.  The  following  describes  the  program  documentation 
required  by  DoD  Instruction  5000.2:  1)  Tte  STAR  is  the  primary  threat  document 
from  which  critical  intelligence  parameters  are  determined.  It  is  initially  prepared  and 
approved  by  the  appropriate  Intelligence  Agency  and  validated  by  the  Defense 
Intelligence  Agency  (DIA);  2)  The  ORD  is  initially  prepared  in  this  phase  by  the  user 
or  the  user's  representative.  The  ORD  reflects  the  criti^  operational  issues  (COIs)  and 
their  criteria  from  which  the  minimum  acceptable  operational  performance 
requirements  are  derived;  3)  The  IPS  is  prepared  at  MS  I.  It  summarizes  the  results  of 
Phase  0.  It  shall  identify  the  most  promising  concepts)  to  be  carried  into  Phase  I  and 
the  reasons  for  elimination  of  alterative  concepts;  4)  A  preliminary  TEMP  containing 
the  requirements  for  T&E,  the  initial  management  responsibilities,  an  outline  of  DT&E 
and  OT&E,  and  resource  requirements  is  prepared.  This  is  the  primary  T&E  document 
for  review  authorities;  and  5)  A  COEA  is  initially  prepared  at  MS  0  and  updated  at  MS 
I.  Significant  here  is  any  change  in  performance  parameters  or  any  adjustment  of 
concepts. 
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4.  EW  SYSTEMS  T&E  DURING  PHASE  I  -  DEMONSTRATIC»^  AND 
VALIDATION  (DEMA^AL) 


4.0.1  Maestone  l-<!oncept  DcmoMtration  Approval.  The  MS  I  decision, 
rendered  in  an  ADM,  selects  the  preferred  EW  concqK(s)  to  contiinK  develq;mient. 
This  concept(s)  will  evolve  into  engmeering  proto^pes  (^)  for  conthmed  T&E.  The 
T&E  outcome  is  the  assessment  of  etch  EP. 

4.0.2  EW  Sv!ttien«  in  PhMe  I  -  DEMA^AL. 

4.0.2.1  Phase  Deseriprion.  The  main  objective  of  diis  phase  is  to  conduct 
technology  demonstrations  and  to  budd  and  test  prmotypes  of  the  EW  s3rstem. 
Requiremmns  are  restated  and/or  updated  in  terms  of  diresholds  and  objectives  and  die 
previously  submitted  preliminary  TEMP  is  updated.  Processes  critical  to  the  most 
promising  system  conoept(s)  should  be  underwood  and  attainable. 

4.0.2.2  TAE  in  DEMA/AL.  Because  DEMA/AL  occurs  early  in  the  EW 
T&E  process,  tests  and  evaluations  may  be  conducted  to  validate  and  qualify  the  design 
and  to  ensure  that  the  product  is  ready  for  Govemmem  accqitance.  Special  emphasis 
should  be  placed  on  the  use  of  HTTL  test  events  to  test  systems  rigorously  in  a  design 
exploration  and/or  refinement  effort  prior  to  Government  validation  testing. 

4.0.2.2.1  Developitiennil  Test  and  Evaluation.  DT&E  conducted  during  the 
DEM/VAL  phase  is  used  to  demonstrate:  1)  that  technical  risk  areas  have  been 
identified  and  reduced  to  acceptable  levels;  2)  that  the  best  technical  approaches  have 
been  accepted;  and  3)  that  from  this  point  on,  engineering  efforts,  rather  than 
experimental  efforts,  are  required.  DT&E  supports  the  MS  n  decision  which  considers 
entry  into  Full-Scale  Development  and,  as  appropriate,  low  rate  initial  production 
(LRIP).  An  EP  can  be  built  in  this  Phase  by  replacing  certain  components  in  the  DSM 
with  hardware  components  developed  during  DEM/VAL. 

4.0.2.2.2  Qpcratioitti  Test  and  F.valuarinn.  OT&E  for  EW  systems  during 
the  DEM/VAL  phase  is  conducted  to  support  the  MS-n  decision  regarding  a  system’s 
readiness  to  move  into  Full-Scale  Development.  As  part  of  OT&E  planning,  the 
operational  aspects  of  the  proposed  technical  approach  are  examined  by  the  OTA. 
Consistent  with  the  evolutionary  requirements  definition,  the  OTA  works  with  the  user 
to  refine  proposed  performance  objectives  and  identify  surge  and  mobilization 
requirements.  Also,  evaluation  criteria  should  be  documented  and  the  test  schedule 
reviewed  for  adequacy  of  time  and  that  the  availability  of  test  articles  is  sufficient  to 
meet  the  OT  objectives.  This  may  include  provisions  for  an  Early  Operational 
Assessment  (EOA).  If  the  MS  II  decision  includes  an  LRIP,  an  EOA  is  required. 
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4.1  STEP  ONE.  Icteitifiatfioo  of  T&E  Inftmnatioii  Required  by  DecisifHi 
Makars  from  T&E.  For  tbe  DEMA^AL  phase,  the  test  requirements  would  have  tym 
identified,  updated,  and  carried  forward  torn  the  phase.  The  TEMP  is 
and  prepared  for  approval  at  MS  n.  If  required,  an  EOA  is  prepared.  The  RCM  and 
the  SMM  are  updated.  Exit  criteria  for  this  phase  are  also 

4.1.1  As  a  minimum.  T&E  addieisws  the  dq>icted  in 

Figure  12  and,  more  specifically  as  listed  below  as  th^r  might  relate  u>  each  EW  EP: 

(1)  Existing  data  analysis 
-  will  the  EP(s)  work? 

What  is  the  confidence 
level  in  this  assessment? 

To  what  extent,  for 
example,  can  the 
situational  awareness  be 
enhanced  by  an  RWR 
EP,  i.e.,  does  it  give  the 
operator  full  four 
quadrant  coverage? 

(2)  Can  the 
requirements,  as  stated, 
be  evaluated? 

(3)  Does  the  existiii^ 

T&E  infrastructure  and 
technology  base  permit 
evaluation?  Rgure  12.  Phase  I  T&E  Questions 

Approximately  what 

is  the  cost  and  time  frame?  If  not,  tqjproximately  what  is  the  cost  and  time 
fiame  to  create  the  tqipropriate  infiastructure  and  technology  base?  If  this  is  a 
software-controlled  EW  are  there  software  modifications  to  be  addressed? 

(4)  What  are  the  capabilities/limitations  of  each  EW  EP  versus  threats?  Is  the 
receiver  sensitiviQr  sufficient  to  detect  low-probability-of-intercept  emitters?  Is 
the  circuitry  sufficiently  hardened  against  high  power  microwaves?  Should  it 
be?  Is  the  tlureat  library  current? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction?  Is 
there  a  moderate-to-high  risk  technology  issue  that  can  be  resolved  early  in  this 
phase? 
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■  (6)  What  alteimtives  exist  to  testing/test  assets?  What  are  the  risks? 

(7)  Has  TEMP  been  approved?  The  TEMP  must  contain  the  performance 
parameters  reflected  in  the  ORD,  COBA,  and  APB.  They  must  be  consistent. 

(8)  Is  each  proposed  EP  feasible?  Is  one  preferred,  and  if  so,  vdiy? 

<9)  What  are  the  recommended  i%ase  n  T&E  "exit  criteria”? 

(10)  What  ate  the  recommended  criteria  certification  of  readiness  for  final 
OT&E? 

(11)  Does  the  EOA  address  the  early  projection  of  potential  operational 
efiectiveness  and  suitability  criteria? 

4.1.2  These  questions  ate  basically  icfentical  to  the  Phase  0  questions.  However, 
because  the  program  is  more  mature,  more  data  at  higher  confidence  levels  will  exist. 

4.2  STEP  TWO.  Pre-test  Analysis.  Because  Pre-test  Analysis  is  more  concerned 
with  the  "how  to”,  or  the  methodology,  of  testing,  the  Pre-test  analysis  Step  in  the 
DEM/VAL  phase  seeks  to  build  on  the  DT&E  actions  from  Phase  0  by  refining  the  test 
trials  and  test  conditions  to  bring  them  closer  to  operational  really.  The  correlation 
between  the  model  and  the  £P  should  be  closer;  aixl,  both  should  be  conveigii^ 
toward  a  match  with  the  operational  requirement.  Development  of  expected  outcomes  is 
again  emphasized.  Future  testing  will  again  be  able  to  take  advantage  of  the  data 
accumulated  during  this  phase. 

4.3  STEP  THREE.  Test  Activity  and  Data  Management.  The  objective  of  the 
T&E  Process  during  DEMA^AL  is  to  gather  data  to  make  trade-ofls  and  to  identify  the 
preferred  t^chnicfll  approach  for  satisfyiig  an  operational  requirement.  In  the 
DEMA^AL  T&E,  an  EP  should  be  tested  to  determine  the  expected  performance  of  the 
proposed  EW  system.  Test  program  planning  should  also  identify  any  required 
upgrades  to  existing  test  resources  or  the  need  for  new  test  resources  in  subsequent 
phases  of  the  program  which  were  not  identified  at  MS  I.  Wherever  practical,  testing  is 
to  be  planned  and  conducted  to  take  full,  cost  effective  advantage  of  existing 
investments  in  DoD  ranges,  facilities,  and  other  resources,  unless  otherwise 
documented  in  the  TEMP.  In  the  DEM/VAL  phase  the  following  are  considered. 

Historical  Test  Data.  Building  on  the  historical  data  search  and  evaluations 
performed  during  Phase  0,  historical  data  in  Phase  I  may  be  extremely  valuable  in 
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developing  confidence  that  the  proposed  development  is  possible  and  in  reducing 
development  risk  by  identifying  successes  and  ^ures  in  earlier  programs. 

Measurement  Tests.  Prototype  hardware  may  be  subjected  to  the  measurement  of 
system  parameters  such  as  antenna  gain  patterns,  receiver  sensitivi^,  ERP,  etc.  This 
type  of  data  is  needed  as  iipit  to  the  analysis  of  system  operation  ami  to  computer 
simulations.  Representative  measurements  that  are  ^ically  accomplished  duiing  Phase 
I  include: 

a.  The  directional  accuracy  of  an  antenna,  its  beamwidth,  and  gain  pattern. 

b.  The  radar  cross  section  (RCS)  of  prototype  lethal  suppression  missiles. 

c.  The  IR  signatures  of  prototype  lethal  suppression  missile  engines. 

Integration  Tests.  It  is  likely  that  DEMA^AL  testing  will  commence  before 
components  are  tested  and  evaluated  by  the  Government.  This  testing  should  focus  on 
identifying  EW  system  hardware  and  software  problems,  maturing  system 
performance,  and  evaluating  estimated  reliabiliQr,  availability,  and  maintainability 
(RAM)  levels.  The  following  are  typical  tests  that  can  be  conducted  at  this  stage  of  the 
process:  receiver  stimulation,  database  cbaractoization,  display  compatibility,  and 
fiiult  testing. 

Hardware-in-the-Loop  Tests.  When  system  integration  laboratory  testing  has  been 
completed,  engineering  prototype  components  should  be  ready  for  HTTL  testily.  Initial 
testing  should  be  conducted  against  a  rigorous  test  environment  and  mature  the  EW 
system  in  a  non-adversarial  manner.  The  main  thrust  of  HTTL  testing  is  to  evaluate  the 
performance  of  actual  hardware  systems/subsystems  and  simulated  environment. 

• 

Installed  System  Tests  on  final  platforms  will  usually  not  occur  in  this  Phase  of 
system  development.  However,  testing  could  be  used  for  equipment  mounted  on  test 
platforms  whem  security  and/or  signal  density  requirentents  can  best  be 
accommodated. 

open- Air/Field  Tests.  The  effects  of  some  real-world  environmental  coiKlitions 
can  be  investigated  only  under  actual  exposure  to  those  conditions.  Therefore,  host 
platform  testing  may  be  required  early  in  an  EW  system's  development  or  modification 
cycle  in  order  to  evaluate  its  achieved  performance.  During  the  DEM/VAL  Phase,  this 
can  be  accomplished  by  installing  the  EW  system  engineering  prototype  in  a  spread 
bench  configuration,  aboard  a  large-body  testbed  aircraft  (or  on  a  higUy  instrumented 
rocket  sled,  when  precision  position  and  velocity  data  are  of  interest),  or  on  a  mobile 
or  fixed  surface  platform  that  would  be  indicative  of  the  "real-world"  environment. 
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Simulatioiis/Srimulatioii  Fvcnts  As  intfae  CE&D  Pbase,  modeling  and 
simulation  should  be  used  in  lieu  of  actual  testing  where  feasible  to  fill  data  voids. 
Hardware-in-the-loop  facilities  and  open-air/field  testing  events  provide  a  csqiability  to 
test  against  man-in-the>loop  threat  simulations  and  sur^:.gate  wei^n  systems.  The 
number  of  people  involved  and  the  complexity  of  the  simulators  can  make  this  testing 
more  expensive  than  in-plant  system  integration  laboratory  testing.  Extensive  pre-test 
analysis  usin^  computer  simulations  should  be  conducted  to  obtain  the  maximum 
benefit  from  sub^^nt  hardware-in-the-loop  and  open-air/field  testing.  The  use  of  a 
mottel  like  ALARM,  or  a  similar  model,  to  assess  technical  performance  slmuld  be 
considered. 

Data  Management.  Data  from  the  test  events  and  trials  in  the  DEM/VAL  phase 
are  processed,  as  in  the  CE&D  phase,  to  convert  raw  data  to  a  form  that  is 
understandable  and  useable  to  decision  makers. 

4.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation  consists  of  reducing  and 
correlatii^  these  measured  data  for  technical  performance  parameters,  measures  of 
effectiveness  (MOEs),  and  measures  of  performance  (MOPs),  on  both  a  quick-look  and 
a  thorough  basis.  This  iterative  assessment  process  begins  with  measurement  testing 
and  continues  through  the  open  air  testing  conducted  during  DEM/VAL.  The  expected 
outcomes  from  Step  Two  should  be  used  in  computer  simulations  to  make  early 
assessments  of  operational  effectiveness  and  suitabiliQr  of  the  EP.  Data  collected  from 
both  the  EW  SUT  and  threat  systems  are  correlated  with  aircraft  flight  paths  as  a 
function  of  time.  Instrumentation  that  monitors  the  electromagnetic  environment 
collects  jammer  and  radar  signal  parameters  and  measures  power  levels.  These  data  are 
compared  with  computer  simulation  or  engineering  estimations.  All  data  are  analyzed 
to  determine  test  results. 

4.4. 1  Results  from  this  Phase  must  be  reported  through  appropriate  channels  to 
the  MDA.  The  results  of  these  tests  and  evaluations  will  have  a  major  impact  on  future 
program  decisions.  Where  required,  OT&E  assessments  are  to  be  reported  at  the  end  of 
each  phase  of  T&E.  The  reports  from  OAs  and  EGAs  are  to  be  submitted  to  the  MDA 
or  PDA  as  required.  These  reports  should  identify  test  limitations  and  their  effects  on 
the  ability  to  demonstrate  whether  or  not  critical  systems  design  issues  and  risk  have 
been  resolved.  Results  may  lead  to  changes  in  specifications  or  may  result  in  a  redesign 
of  the  system. 

4.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
Program  Information.  The  ouq^ut  of  this  step  is  an  informed  decision  on  the  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 
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4.5.1  Milestone  n  Decision  Points.  During  the  Milestone  II  review,  the 
MDA/PDA  reviews  the  results  to  confirm  that  they  provide  reasonable  assurance  that 
the  techimlogies  and  processes  critical  to  successful  development  of  die  EW  system  are 
attainable  and  that  the  threat  assessment  and  mission  need  are  still  current  and  valid. 
The  LRIP  quantities,  the  exit  criteria  for  Phase  n,  and  the  Developmental  baselii^  are 
all  approved,  and  ^roval  is  granted  to  enter  into  Phase  II. 

4.5.2  nevelopmentfll  Baseline.  Compared  to  the  Concept  Baseline,  the 
Developmental  Baseline  contains  more  detailed  and  refined  cost,  schedule,  and 
performance  objectives  and  thresholds.  Tte  list  of  system  and  technical  performance 
parameters  should  be  expanded  from  the  key  parameters  of  the  Concept  Baseline  that 
were  derived  fiom  testing  the  engineering  prototype.  Performance  objectives  set  for 
Phase  n  should  represent  a  realistic,  meaningful,  measurable,  cost-effective,  and 
affordable  incremem  in  operational  capability  beyond  the  minimum  acceptable 
requirements. 

4.5.3  Dncumentation.  The  following  is  the  program  documentation  required  by 
DoDI  5000.2:  1)  The  STAR  is  updated  at  MS  n  and  at  other  points  in  the  program 
determined  by  the  MDA.  It  reflects  new,  validated  threat  assessments,  as  well  as 
addresses  any  unresolved  threat  concerns  and  critical  intelligence  parameters  that  have 
surfaced  during  DEMA^AL  T&E;  2)  The  ORD  is  updated  and  expanded  for  MS  II  to 
include  thresholds  and  objectives  for  mote  detailed  and  refined  performance  capabilities 
and  characteristics  based  on  the  results  of  trade-off  studies  and  testing  done  during 
Phase  1.  Key  parameters  from  the  ORD  are  included  in  the  Development  Baseline  at 
MS  II;  3)  The  IPS  at  MS  II  summarizes  the  results  of  Phase  I  and  how  the  exit  criteria 
in  the  MS  1  ADM  were  satisfied.  It  also  identifies  further  risk  reduction  efforts,  trade¬ 
off  decisions,  a  summary  of  cost  estimates  and  assessments,  and  the  proposed 
acquisition  strategy,  proposed  waivers,  any  LRIP  quantities  and  the  T&E  events  to  be 
accomplished  prior  to  LRIP  contract  award;  4)  The  TEMP  is  finalized  and  contains 
data  necessary  to  conduct  DT&E  and  OT&E.  DT&E  events  follow  the  conduct  of  test 
events  and  are  labeled  as  DTI,  DTIA,  DTIB,  etc.  OT&E  events  are  labeled  in  a  similar 
manner.  The  update  should  describe  testing  performed  to  date  and  the  attainment  of 
thresholds  and  objectives  of  performance  parameters.  The  SMM  and  the  RCM  are 
updated;  and  5)  the  COEA  at  MS  II  addresses  the  most  promising  system  concept 
demonstrated  and  validated  in  Phase  H.  Performance  and  cost  intervals  should  be 
narrowed  to  point  estimates.  A  sensitivity  analysis  is  performed  to  identify  any  critical 
sensitivities  of  the  EW  system's  effectiveness  to  test  restrictions,  such  as  safety 
constraints  or  test  resource  limitations. 
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5.  EW  SYSTEMS  T&E  DURING  PHASE  H  -  ENGINEERING  AND 
MANUFACTURING  DEVELOPMENT  (EMD) 


S.0.1  Milestone  n-Develnpmental  Approval.  The  MS  11  decision  detemunes  the 
progress  of  the  selected  EP(s)  for  continued  development  as  production  prototypes 
(PPs).  The  T&E  outcome  is  the  assessment  of  each  selected  prototype. 

S.0.2  EW  Systems  in  Phase  n  -  EMD. 

S.0.2.1  Wiflse  Description.  The  main  objective  of  this  phase  is  to  tksign, 
fiU)ricate,  test  and  evaluate  an  EW  PP;  to  validate  the  manufiu:sunng  or  production 
process  confirming  that  design  risks  have  been  mitigated;  and  to  demonstrate  thitmgh 
testing  that  the  system  capabilities  meet  specificationrequirements  and  satisfy  mission 
needs. 


5.0.2.2  T&E  in  EMD.  The  main  objectives  of  T&£  in  EMD  are  to  ensure  that 
engineering  is  conq)lete,  to  validate  system  performance  of  the  production  design  in  a 
realistic  environmeiu,  and  to  ensure  that  specifications  are  met  through  preproduction 
and  production  qualification  testing.  Further,  a  test  plan  is  developed  to:  ensure  design 
problems  are  solved;  provide  the  guidelines  for  software  testii^;  validate  configuration 
changes;  validate  system  compatibility  and  interoperability  with  other  EW  systems 
and/or  other  platform  electronic  systems;  and  continue  the  reliability,  availability,  and 
maintainability  program. 

S.0.2.2.1  Developmental  Test  and  Evaluation.  DT&E  conducted  during  this 
phase  provides  the  final  technical  data  to  determine  the  EW  system's  readiness  to 
transition  into  either  LRIP  or  full-rate  production.  DT&E  in  EMD  is  conducted  with 
prototype  hardware  and  is  characterized  by  the  use  of  engineering  and  scientific 
approaches  under  controlled  conditions  to  provide  quantitative  and  qualitative  data  for 
use  in  the  system's  evaluation.  Technical  performance  is  measured  with  respect  to  both 
operational  effectiveness  and  suitability  factors  that  include  reliability,  availability, 
maintainability,  compatibility,  interoperability,  safety,  and  supportability.  The 
implementing  command  must  certify  that  the  EW  system  is  ready  for  lOT&E. 

5.0.2.2.2  Operational  Test  and  Evaluation.  The  primary  purpose  of  EMD 
OT&E  is  to  support  a  fall-rate  production  decision  at  Milestone  HI.  An  OA  may  be 
required  to  support  LRIP  of  the  system.  After  the  implementing  command  certifies  that 
the  system  is  ready  for  OT&E,  the  OTA  conducts  lOT&E  on  a  representative 
production  system  to  evaluate  operational  effectiveness  and  suitability  tests  and  to 
ensure  that  the  system  meets  its  minimum  operational  thresholds.  lOT&E  also 
addresses  logistics  and  software  support  requirements,  identifies  deficiencies  or  the 
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need  for  modl&ations,  and  provides  information  to  refine  training,  tactics, 
countenneasure  techniques,  and  doctrine. 

5.1  STEP  ONE.  Idoitificaticni  of  T&E  Infonnation  Required  By  DedsiiNi 
Makers  iktau  T&E.  The  main  objective  of  the  Process  in  EMD  is  to  ensure  that  the 
defined  and  selected  operational  effectiveness  and  suitability  performance  parameters, 
along  with  design,  specification,  and  production  characteristics,  are  still  valid  for  the 
EWSUT. 


S.1.1  As  a  miniinum,  T&E  addresses  the  (generic)  questions  as  dqncted  in 
Hgure  13  and,  more  specifically 
as  listed  below  as  they  might 
relate  to  each  EW  PP: 

(1)  For  DT&E,  what 
are  the  capabilities 
and  limitations  of 
each  PP  being 
developed?  What  is 
the  co^dence  level  in 
these  data  and/or  this 
assessment? 

(2)  For  OT&E,  are 
the  PPs  suitable  and 
effective  in  satisfying 
the  mission  need? 

What  is  the 
confidence  level  of 
this  assessment?  Are  P*8»*re  13.  Phase  H  T&E  Questions 

the  PPs  "user 

friendly”,  i.e.,  are  typical  users  exposed  to  the  SUT  and  their  reactions  to 
system  operations  considered? 

(3)  Have  key  performance  objectives/threshoids  been  validated  versus 
advanced  threats?  Are  the  simulated  threats  sufficiently  representative  both 
in  numbers  and  capability? 

(4)  Are  the  PPs  feasible?  Do  they  satisfy  the  need?  Is  one  preferred?  Is  the 
design  stable  and  producible? 


Phaw  n  -  Engiiiwriiig  St  Manufacturing  Oendopment 
For  odi  prnfyetkM  praoijrpe  (FT). 

(t)  For  OTAE.  wtat  wc  the  capoMlitka  ond  limiaiiara  of  cteh  PP  beini 
devctoped?  Wtet  it  the  caafideoce  level  in  ihit  dua/ttMtanent? 

(2)  FOr  OTAE.  tie  die  PPt  luitable  ind  cfllKtive  in  mitiyint  die  mniion 
neod?  Wlmitditconftdencolevciofildt 

(3)  Have  key  petfannance  otjenivet/difMlioidi  been  validated  vertut 
advanced  drait? 

(4)  Ace  die  PPt  feaiiMc?  Do  diey  taiiUy  die  need?  It  owe  prefeited? 

(5)  Have  requcemeMa  ciiantea  been  incocpotated  into  die  APB.  ORD  and 
oonoea  HKcificaiioet? 

(ti)  Have  tpeciflcaiioo  dianiBt  been  reflected  back  to  rcqiiiicinenct  and 
inoocporaKd  mo  dw  APB  and  ORD? 

(7)  Doca  die  TEMP  reflect  the  cfaaagea  in  (S)  and  (6)?  Hat  it  been  approved? 

(S)  Should  there  be  a  Milettane  IV.  wbu  are  die  recommended  Ptiase  ID  TAE 
'cakeriieeia'? 
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(5)  Have  requiremeitts  changes  been  incorporated  into  the  APB,  ORD,  and 
contract  specifications? 

(6)  Have  specification  changes  been  reflected  b«dt  into  the  requirements 
and  incorporated  into  the  APB  and  ORD? 

(7)  Does  the  TEMP  reflect  the  changes  in  (5)  and  (6)?  Has  it  been 
approved? 

(8)  Should  there  be  a  MS  IV,  vAat  are  the  recommended  Phase  OX  T&E 
"exit  criteria"? 

5.2  STEP  TWO.  Pre-test  Analysis  is  conducted,  as  required,  to  analyze  test  trial 
conditions  or  establish  expected  outcomes  before  progressing  through  an  orderly  test 
process  beginning  with  system  integration  laboratory  (SIL)  T&E  and  ending  with 
field/open-air  range  testii^.  Evaluation  actions  in  this  Phase  are  contituous  and  begin 
with  the  Pre-test  analysis.  Computer  aided  simulation  analysis  may  be  used  to  assess 
the  expected  performance  of  the  EW  syston  under  various  test  conditions  and  to 
complete  any  remaining  pretest  planning  factors.  In  addition,  the  DSM,  if  used,  is 
updated  to  support  future  evaluation  of  engineering  change  proposals  (ECPs)  and 
modifications. 

5.3  STEP  THREE.  Test  Activity  and  Data  Management.  EMD  T&E  is 
conducted  to  determine  whether  the  EW  systmn  meets  all  critical  operational 
requirements  when  it  is  produced  in  a  cost  ef^tive  manner.  It  is  significant  that  the 
manufacturing  and  production  process  is  validated  at  this  point,  and  it  represents  a  final 
opportunity  for  mo^flcations  to  be  made  to  tte  system  detign  before  the  EW  system 
enters  final  production.  T&E  conducted  is  commensurate  with  the  capabilities  required 
for  that  event.  In  the  EMD  Phase  the  following  are  considered. 

Historical  Test  Data.  Durii^  EMD,  historical  data  searches  can  fulfill  several 
functions:  1)  Historical  data  may  obviate  the  need  for  certain  tests,  2)  It  may  identify 
previously  unsuccessful  designs  and  either  suggest  improvements  or  steer  the  program 
clear  of  pitfalls,  and  3)  It  may  suggest  or  provide  answers  to  questions  raised  when 
actual  performance  of  the  current  SUT  does  not  meet  expectations. 

Component  Measurement  Test  events  at  this  stage  are  much  more  likely  and 
useful.  With  the  EW  system  installed  on  projected  host  platforms,  valid  measurements 
can  be  made  which  confirm  design  capabilities,  identify  design  deficiencies,  and 
determine  employment  options.  Measurement  testing  should  establish  values  for 
technical  performance  parameters  for  installed  antenna  patterns,  platform  signatures. 
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direction  finding,  conqwnent  reliabiliQr  under  various  field  tests,  and  decoy 
trajectories. 

Tnteyrarion  Tests.  Similar  to  DEMA^AL  testing,  the  EMD  testing  may  begin  in 
a  test-fix-test  sequence.  An  electromagnetic  environment  generator  can  be  used  to 
support  this  testing  by  stimulating  components  of  the  engineering  prototype  to  evaluate 
performance  and  compliance  with  technical  requirements.  This  testing  should  confirm 
tibat  performance  thresholds  ctqiable  of  being  tested  in  the  integration  laboratory  have 
been  achieved  and  correct  aiiy  identified  hardware  and  software  problems.  Once  the 
cmi^onems  of  the  SUT  have  been  tested  and  evaluated  they  can  be  prepared  for  more 
rigorous  evaluation. 

Hardware-In-The-Loop  (HTTL^  Tests.  The  production  prototype  components  are 
usually  tested  in  Government  HTTL  test  facilities  before  the  system  is  installed  in  a 
testbed  for  field/open-air  testing.  This  ground  testing  should  focus  on  confirming  that 
identified  problems  have  been  fixed,  that  performance  thresholdscan  be  achieved,  and 
on  optimizing  countermeasure  techniques  versus  HTTL  threat  simulators.  Because  a 
complete  EW  system  is  available  for  this  testing  (instead  of  just  critical  components  as 
in  DEMA^AL  testing),  this  is  the  first  opportuni^  to  conduct  integrated  system 
effectiveness  tests.  Specific  tests  to  be  conchicted  depend  on  the  functions  included  in 
the  system.  Some  examples  are: 

a.  Testing  the  cs{)ability  of  radio  frequency  (RF)  warning  receivers  to 
process  a  high-density  signal  environmexit. 

b.  Optimizing  RF  countermeasures  versus  manned  hybrid  threat 
simulators  and  demonstrating  achieved  effectivemss  in  terms  of  tracking  errors  and 
missile  miss  distances. 

c.  Determining  the  effectiveness  of  RF  countermeasures  versus  early 
waming/GCI  radars  and  communication  links. 

Installed  System  Tests  are  normally  the  first  opportunity  to  evaluate  system 
operation  on  a  weapon  system  platform.  They  are  conducted  to  evaluate  the  integrated 
performance  of  EW  subsystems  as  part  of  a  weapon  system  platform.  A  prime  purpose 
is  to  test  the  electromagnetic  compatibility  (EMC)  of  the  EW  SUT  with  other  systems 
on  the  host  platform.  Some  examples  are: 

a.  Stimulate  the  EW  system  with  mission  representative  threat  signal 
environment  and  verify  correct  display  when  communication,  navigation,  and 
identification  avionics  systems  are  transmitting. 
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b.  Confinn  that  RF  receivers  can  detect  and  identify  threats  in  specified 
times  while  blanked  for  jammer  transmissions. 

c.  Measure  jammer  duty  cycle  m  ensure  it  meets  specifications. 

Field/Qpen-Air  Tests  present  tte  first  opportunity  to  measure  selected 
performance  parameters  of  the  EW  system  in  the  actual  operating  environment  of  its 
host  weapon  system  platform.  They  provide  the  means  to  calibrate  the  crther  classes  of 
events  (i.e.,  digital  simulations,  system  integration,  HTTL,  installed  system  tests)  and 
to  validate  the  expected  (nitcomes  and  the  measured  outcomes  thereby  establishing  an 
acceptable  confidence  factor.  Flight  testing  normally  begins  widi  one-onone  scenarios 
and  progresses  to  multipie  threat  system  scenarios  constructed  to  be  as  opmtionally 
realistic  as  possible,  using  available  threat  simulator  resources,  ranges,  and  range 
instrumentation.  Some  examples  of  EMD  open-air  range  testily  are: 

a.  Conduct  pod  stores  release  certification  tests  for  pod-mountod 

systems. 


b.  Confirm  that  threat  signals  are  detected  at  specified  ranges  and 
correctly  idemified. 

c.  Determine  if  countermeasures  degradation  of  threat  perfonnajtx:e 
parameters  meets  required  thresholds. 

For  lOT&E,  field/open-air  testing  is  indispensable  for  evaluating  operational 
effectiveness  aixi  suitability  because  it  provides  the  final  basis  for  conq>arison  of 
previously  collected  data  and  a  point  of  departure  for  additional  simulation,  analysis, 
and  evaluation. 

5»iTnulations/.Stimulation  Events.  As  in  the  DEMA^AL  I%ase,  modeling  and 
simulation  should  be  used  in  EMD  in  lieu  of  actual  testii^  where  feasible  to  fill  data 
voids.  Computer  simulations  should  be  run  to  support  pre-test  analysis  in  order  to 
structure  new  test  trials  and  update  the  values  expected  for  system  performance  and 
technical  performance  parameters.  The  use  of  models  like  AASPEM  and 
SUPPRESSOR,  or  similar  models,  during  this  phase  should  be  considered.  Both 
AASPEM  and  SUPPRESSOR  are  appropriate  for  assessing  operational  performance. 

Data  Management.  Data  (e.g.,  antenna  patterns,  time  responses,  tracking, 
telemetry,  firing  events,  operator  logs)  measured  in  various  test  facilities  must  be 
processed  into  a  form  suitable  for  analysis.  The  actual  data  may  be  understaixlable 
only  to  the  analyst.  This  is  particularly  true  during  the  early  forms  of  testing  when 
DSMs  may  be  used  more  than  the  actual  hardware.  In  this  step,  the  data  are  mmed  into 
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meaningful  infonnation  which  can  be  understood  by  the  average  person  uiio  has  some 
familiarity  with  the  system  and  the  test  requirements.  Test  managers  must  work  closely 
with  the  test  force  and  test  range/facility  personnel  to  determine  what  data  are  collected 
and  the  required  reporting  formats  for  the  data.  Once  processed,  time  correlated 
measurement  values  can  be  used  to  conduct  a  post-test  assessment  of  the  results. 

5.4  STEP  FOUR.  FOst-test  Synthesis  and  Evaluation.  As  higher-confidence  test 
data  fiom  the  EW  PP  become  available,  post-test  synthesis  and  evaluation  focuses  on 
establishing  demonstrated  values  for  key  system  perficnmanoe  parameters  and  critical 
technical  performance  parameters.  At  this  point,  test  data  permits  a  more  precise 
statistical  assessment  of  actual  system  performance.  Test  data  from  lOT&E  related 
activities  are  analyzed  to  determine  system  operational  effectiveness  and  suitability. 

This  aids  in  establishing  confidence  levels  for  the  values  and  in  the  development  of 
algorithms  for  future  applications.  Any  developed  simulation  or  model  must,  with  the 
use  of  this  analysis,  be  updated.  Coupling  the  measured  outcomes  of  testing  with  those 
expected  from  simulations  and  models  provides  an  opportunity  to  revalidate  the 
simulations  and  models  and  update  their  parameters  to  improve  the  model's  accuracy. 
Models  can  be  used  to  evaluate  the  test  data  in  terms  of  military  needs  and  operational 
requirements.  Feedback  to  the  decision  makers  with  infonnation  needed  to  assess  the 
system's  military  worth  and  program  viability  is  key  to  the  process.  The  results  of 
T&E  in  terms  of  assessments  must  be  provided  to  the  PM,  PEO,  MDAs,  and  other 
offices  as  required.  This  is  accomplished  through  DT  and  OT  final  reports  and 
briefings.  This  reporting  enables  the  DoD  EW  T&E  Process  to  provide  analyzed  and 
correlated  data  in  understandable  terms  for  a  valid  program  assessment.  In  addition  to 
the  final  rqmrts,  the  feedback  loop  also  includes  archiving  other  relevant  measured 
data  and  outcomes  along  with  the  test  reports  to  facilitate  the  maintenance  of  a 
thorough  history  (audit  trail)  of  the  program.  This  archivii^  process  also  includes 
updating  the  SMM  by  indicating  the  measuied  outcomes  resulting  from  testing  the  EW 
production  prototype. 

5.5  STEP  FIVE.  Decision  Maker  Wdghs  T&E  Information  Against  Otbor 
Program  Infonnation  to  decide  a  proper  course  of  action.  When  associated  with  an 
acquisition  milestone,  this  decision  is  announced  in  an  ADM  which  outlines  the  future 
course  of  action  for  the  program  and  the  SUT.  Additionally,  each  milestone  decision 
contains  the  "exit  criteria"  for  the  next  Phase/Milestone. 

5.5.1  Milestone  HT  Decision  Points.  T&E  results  will  be  reported  by  the 
responsible  test  organization.  The  DT&E  report  compares  assessed  values  of  critical 
system  performance  parameters  with  threshold  levels  specified  in  the  Phase  II 
Acquisition  Program  Baseline  (i.e.,  RCM  and  SMM).  It  also  assesses  the  readiness  of 
the  system  for  lOT&E  and  production.  The  OTA  prepares  final  lOT&E  reports  that  are 
timely,  factual,  concise,  complete,  accurate,  aixi  objective.  The  final  repon  assesses  the 


38 


adequaqr  of  conducted  OT&E,  and  wliedier  the  test  and  evahiadon  xesuUs  conilnn  that 
items  or  components  tested  are  c^ierationaUy  effective  and  suitable  for  use  in  cmnbat  by 
typical  military  users. 

5.5.2  Th^  Pmrfiiprinn  Rg«i>lwM»  rinriiwiMW*  th^  final  paffnimance  th«»«hnlHt  aiiH 
objectives,  as  well  as  costs  and  schedules,  for  the  I^oduction  and  Dq)loynient  Phase  of 
the  program.  At  Milestone  m,  performance  thresholds  in  the  system  devdopment 
product  specification  should  be  traceable  to  die  performance  duesholds  and  objectives 
documented  in  the  Production  Baseline.  These  values  are  approx  by  the  MDA  at 
Milestone  m. 

S.5.3  TVicutnenfatinn.  T.isied  below  is  the  program  documentation  required  by 
DoD  Instruction  5000.2  that  should  be  reviewed  prior  to  this  Phase  and  shodd  contain 
the  latest  iqxlates:  1)  The  STAR  is  tqidated  with  emphasis  on  changed  and/or  updated 
critical  intelligence  parameters  that  may  be  general^  as  new  requirements,  2)  The 
ORD,  after  MS  n,  should  be  modified  only  as  a  result  of  changes  in  the  MNS  or  cost- 
schedule-performance  trade-offe  during  Pl^  n.  Key  parameters  from  the  ORD  are 
included  in  the  Production  Baseline  at  MS  IB.  The  TAdE  mana^rs  are  immediately 
notified  of  ai^  changes  to  ensure  that  system  T&E  can  be  administered  without  delay 
and  that  the  required  T&E  strategy  still  enables  the  SUT  to  render  the  acceptable 
degree  of  performance  validity,  3)  The  IPS  is  updated  to  describe  program  changes 
since  MS  II  and  how  the  exit  critmia  in  the  MS  n  ADM  were  satisfied.  It  will  also 
address  risk  reduction  efforts,  any  proposed  preplanned  product  improvement  (P3D 
recommendations,  program  cost  estimates  and  assessments,  and  the  proposed 
acquisition  strategy  for  the  remainder  of  the  program,  4)  The  TEMP  is  updated,  as 
required,  and  updates  reflect  testing  completed  and  demotistrated  values  attained  to 
date.  The  SMM  and  RCM  are  updated  and  emphasis  is  placed  on  completion  of 
DT&E,  its  certification,  and  progress  towards  lOT&E/FOT&E,  5)  At  MS  IB,  a  COEA 
is  not  required  unless  conditions  have  changed  sufficiently  so  that  previous  cost 
effectiveness  determinations  ate  no  longer  valid.  MS  IB  analyses  often  only  provide 
updated  estimates  of  life  cycle  costs.  If  a  change  is  of  sufficient  magnitude  to  cause  the 
DAB  to  revisit  its  MS  B  decision,  the  full  MS  B  COEA  is  updated.  The  T&E  concern 
here  is  changes  to  requirements  which  affect  T&E  events  and  actions,  6)  OT&E 
Report.  This  additional  document  is  required  by  law  to  be  submitted  to  the  Secretary  of 
Defense  and  Congress  for  major  acquisition  programs.  This  is  a  DOT&E  action. 
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6.  SYSTEMS  T&E  DURING  PHASE  HI  -  PRODUCTION  AND  DEPLOYMENT 
(P&D) 


6.0.1  Milestone  m-Prodiictiott  Approval.  The  MS  m  decision  detennines 
Dviiether  to  enter  production  and  deploy  the  EW  systems  under  devek^nnent. 

6.0.2  EW  System  TfeR  in  Phase  OT  -  PAH, 

6.0.2. 1  Phase  Description.  The  main  obiective  of  the  P&D  phase  is  to 
ensure  that  an  efGknent  and  stable  production  i»ocess  is  established  with  an  adequate 
technical  suqppoit  base,  and  to  ensure  that  the  user  is  sausfied  that  die  iaended  EW 
misskin  need  has  been  met.  Odier  objectives  are  to  conduct  foUow-cm  operational  and 
proditt^tion  verification  testing  to  confirm  and  monitor  performance  and  quality  and  to 
verify  the  correction  of  deficiencies.  If,  during  production,  a  requirement  for  a  major 
modification  is  identified,  an  additional  milestone.  Milestone  IV,  may  be  required.  It 
will  be  scheduled  to  review  and  approve  ar^  proposed  modifications  to  the  acquisition 
strategy,  the  program  plan,  and  APB  (concqn,  development  or  production)  and 
establish  the  exit  criteria  that  must  be  met.  If  the  system  is  no  longer  in  production,  the 
need  for  an  iqpgrade  must  be  established  through  a  new  MNS. 

6.0.2.2  TAP,  in  PAD.  The  objectives  of  T&E  in  P&D  are  to  demonstrate 
an  operational  capabiliQ^  of  the  production  EW  system  that  satisfies  the  mission  need 
and  to  identify  additional  follow>on  operational  test  and  evaluation  (FOT&E)  that  may 
include  follow-on  operational  and  inoduction  verification  testing  in  accordance  with  the 
TEMP. 


6.0.2.2.1  Develnpmental  TAP  is  cnnducted  to  confirm  that 
specifications  are  being  met  on  the  production  version  of  the  system  and  to  evaluate 
any  product  improvement  chaises,  such  as  those  resulting  from  ECPs,  changes  in  the 
threat,  efforts  to  reduce  system  life  cycle  costs,  or  efforts  to  improve  system  reliability, 
maintainability,  and  availfoilify.  This  is  normally  accompliitiied  through  a  limited 
number  of  ground  and  flight  tests.  If  no  chaises  are  made  on  the  initial  production 
systems,  there  should  not  be  many  additional  DT&E  requirements. 

6.0.2.2.2  Operational  TAP  conducted  on  the  production  version  of  the 
system  is  usually  tem^  FOT&E.  FOT&E  evaluates  operational  effectiveiwss  and 
suitability,  examines  modifications  or  changes,  and  determines  whether  the  EW  sysmm 
m^ts  operational  and  logistic  support  needs  in  changing  operational  environments.  It  is 
also  used  1)  to  refine  estimates  of  operational  effectiveness  and  suitability  made  during 
lOT&E,  2)  to  evaluate  changes  made  to  correct  deficiencies  found  in  prior  T&E,  and 
3)  to  identify  potential  additional  deficiencies.  FOT&E  is  conducted  throughout  the 
remainder  of  the  system's  life  and  further  evaluates  the  system's  operational 
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e£fectivaiess  and  suitability  to  update  training,  tactics,  techniques,  and  doctrine  and  to 
idoitify  potential  deftciencies  or  the  need  for  modifications. 


6.1  STEP  ONE.  Idend&adon  of  T&EInfamiatioo  Required  By  Dedtdon 
Makffs  from  T&E.  The  main  objective  in  this  phase  is  to  refine  diose  EW  system 
concept(s),  (operational  effectiveness  and  suitability  performance  parameters,  along  widi 
design,  spec^cadon,  and  production  parameters  that  must  be  tes^  to  ensure  that 
potential  deficiencies  and  tect^nized  modifications  can  be  corrected  or  applied  to  the 
system.  The  SMM  and  RCM  are  updated  widt  any  applicable  changes  to  threshohls  and 
ob^cdves  of  oitical  performance  parameters  diat  must  be  accomplished  during  Phase 

m. 


6.1.1  Asa 

minimum,  T&E  addresses 
the  (generic)  questions  as 
depicted  in  Figure  14  and, 
more  specifically,  as 
listed  below  as  they  might 
relate  to  each  EW  sj^tem: 


(1)  Fbr 
DT&E,  what 
are  the 
deployed 
system's 

capabilities  and  Rgure  14.  Phase  m  T&E  (Questions 

limitations?  What  ate 

the  capabilities  and  limitations  of  modifications  and  upgrades? 

(2)  For  OT&E,  does  the  EW  system  continue  to  be  operationally  suitable 
and  effective  in  operational  use?  Do  proposed  modifications  and  upgrades 
increase  the  suitability  and  effectiveness  of  the  system? 

(3)  Have  performance  objectives  and  thresholds  against  advanced  threats 
been  validated?  Does  the  system  meet  these  performance  objectives  and 
thresholds? 


(4)  Is  the  TEMP  current? 

(5)  Should  a  MS  rv  be  directed  after  MS  HI  and,  if  so,  what  are  the  Phase 
rV  T&E  "exit  criteria"? 
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6J.  Sl'EP  TWO.  Pretest  Analysis.  Any  changes  in  the  hardware  and  software  at 
this  point  in  a  program  often  require  additional  testing  to  voify  and  validate.  As 
before,  the  process  begins  with  pre-test  analysis  to  evaluate  changes  u>  the  baseline 
necessary  either  to  correct  noted  deficiencks  or  to  modify  the  system  in  response  to 
threat  changes.  The  data  produced  by  this  analysis  axe  us^  to  establish  higter 
confidence  cjqMxted  values.  The  results  continue  to  be  used  fo  support  program 
decisions,  and  the  DSM  is  iqxiated  to  reflect  any  changes  in  demonstra^  performance. 

63  STEP  THREE.  Test  Activity  and  Data  Management.  Testing  develops  the 
Hata  to  an  operational  capability  for  the  production  system  that  satisfies  the 
mission  need  and  idaidfies  any  additional  required  FOT&E.  In  the  P&D  Phase,  the 
following  are  considered. 

Historical  Test  Data.  During  P&D,  the  use  of  historical  data  is  critical.  Archiving 
system  data  in  historical  files  allows  evalmuion  of  the  nature  of  an  observed  deficiency. 
If  it  is  observed  only  at  a  single  location,  the  problem  is  probably  not  systemic  and  tlK 
search  for  solutions  should  first  explore  possible  causes  at  that  location.  If  the  problem 
is  widespread,  a  thorough  review  of  the  nuances  of  the  deficiency  may  point  more 
clearly  to  the  source  of  the  problem  and  may  suggest  or  even  dictate  the  solution  or 
may  prevent  returning  to  a  previous  design  that  was  earlier  proven  unacceptable. 

rnmponent  Measurement  Tests.  Changes  in  the  system's  antennas,  in  antenna 
locations  on  platform(s),  or  in  the  host  platform  signature,  require  additional  static 
measurements.  Measurements  are  also  coiKlucted  during  field  tests  to  obtain  dynamic 
signature  and  antenna  pattern  parameters. 

Integration  Tests.  The  amount  of  SIL  testing  i»cessary  in  this  phase  depends  on 
the  number,  extent,  and  complexity  of  changes  to  the  system. 

Hardware-ln-The-Loop  (HTTL^  Tests.  If  modifications  are  made  to  the  system 
prior  to  or  during  production,  additional  HTTL  testing  may  be  required.  Changes  to 
countermeasure-response  functions,  in  particular,  usually  require  additional  HTTL 
testing  to  verily  that  the  proposed  responses  are  effective. 

Installed  System  Tests.  Installed  system  testing  continues  during  the  P&D  Phase 
to  evaluate  the  installed  system's  interfaces  and  interoperability  with  other  platform 
systems.  The  actual  configuration  of  the  system  to  be  deployed  is  tested  in  installed 
system  tests  before  testing  on  open-air  ranges.  This  procedure  helps  in  identifying 
problems,  saves  test  hours,  and  produces  more  usable  test  runs. 

Field/Open  Air  Tests.  Open  air  range  testing  is  often  necessary  to  determine  if 
the  production  configurations  of  the  system  satisfy  user  requirements.  This  testing  may 
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include  new  threat  simulators  that  are  nk>re  rqjresentative  of  the  actual  threat  and  may 
en^loy  expanded  test  scenarit^  as  more  test  assets  become  available.  These  tests 
should  conflnn  wh^her  systrai  performance  requirements  estid>lisfaed  in  previous 
phases  have  been  achieved. 

Simulatinns/Stimulation  Events.  As  in  the  EMD  Phase,  modeling  and  simulation 
should  be  used  in  P&D  in  lieu  of  actual  testing  where  feasible  to  fill  data  voids.  Wtum 
new  test  scmiaiics  are  developed  to  respond  to  changes  in  requirements,  computer 
simulations  should  be  used  to  design  new  test  trials  and  estimate  test  results. 

Data  Management.  Data  are  processed  into  meaningful  information  that  can  be 
understood  by  a  person  who  has  some  familiarity  with  the  system  and  the  test 
requirements.  Once  processed,  time  correlated  measurement  values  can  be  used  to 
coquet  a  post-test  assessment  of  the  results. 

6.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation.  As  higher  quality  test  data 
from  production  systems  become  available,  post-test  synthesis  and  evaluation  focuses 
on  establishing  demonstrated  values  for  key  system  performance  parameters  and  critical 
technical  performance  parameters.  At  this  point,  test  data  permit  a  more  precise 
statistical  analysis  of  a^eved  system  performance  and  effectiveness  and  establishes 
confidence  levels  for  performance  paranwter  values. 

The  data  provided  as  feedback  are  presented  in  the  form  of  assessments  that  aid  the 
decision  m^rs  in  reaching  their  decisions  on  the  progress  of  the  SUT.  Full  rate 
production  of  the  system  is  achieved  during  the  Pd^  Phase.  Before  full-rate  production 
is  initiated,  however,  open  ECPs  should  be  evaluated  and  resolved.  ECPs  may  also  be 
generated  due  to  changes  in  threat  and  cost  projections,  or  to  achieve  reliability  and 
maintainability  growth  goals.  Full-Rate  Production  incorporates  the  changes  generated 
by  approved  ECPs  during  the  EMD  Phase.  These  changes  are  installed  on  new  host 
platforms  to  establish  the  initial  deployment  of  the  EW  system.  Frequently,  a  number 
of  the  full-rate  production  systems  are  used  as  test  items  for  DT&E  and  FOT&E. 

6.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
^ogram  Information.  The  output  of  this  step  is  an  infoimed  decision  on  the  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 

6.S.1  Milestone  IV  Decision  Points.  Reports  from  DT&E  activity  and  a  full 
report  of  OT&E  results  are  prepared  and  presented  at  the  conclusion  of  T&E  events 
and/or  actions.  These  reports  support  program  decisions  and,  if  applicable,  the  MS  IV 
review.  During  the  P&D  Phase,  on-going  mission  area  analyses  may  identify  threat 
changes,  or  follow-on  operational  testing  may  identify  system  deficiencies,  that  require 
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modificatioiis  to  the  Systran.  Modifications  could  also  result  from  changes  in  defense 
planning  guidance  or  from  of^itunities  to  reduce  ownership  costs.  If  this  is  the  case 
and  the  system  is  still  in  production,  an  MS  IV  review  may  be  requited  to  ensure  that 
reasonable  alternatives  have  been  thoroughly  examined  prior  to  commitment  to  a  major 
modification  program. 

At  an  MS  IV  review,  the  MDA  confirms  that:  1)  the  threat  assessment  and 
performance  objectives  and  thresholds  for  the  modification  have  been  validated,  2)  field 
raq)erience  and  results  support  the  need  for  such  a  program,  3)  a  reasonable  assurance 
exists  that  the  technologies  and  processes  critical  to  success  have  berai  identified  and 
are  attainable  in  the  context  of  the  acquisition  strategy  or  phase  being  pr(^K>sed,  and  4) 
the  program  is  affordable  and  resources  are  available  to  support  the  program. 

The  ADM  resulting  from  an  MS  IV  review  to  approve  a  permanent  modification 
program  has  to:  1)  define  the  phase  of  the  acquisition  process  the  program  is  approved 
to  enter;  2)  approve  the  proposed  or  modified  acquisition  strategy  and  baseline  for  the 
program  (e.g.,  concept,  development  or  production);  and,  3)  establish  program-specific 
exit  criteria  that  must  be  accomplished. 

6.5.2  DficuTnentation.  Each  document  is  to  be  updated  as  follows  in  accordance 
with  results  from  EMD  testii^  performed  to-date:  1)  llie  STAR  is  updated,  if  required, 
at  MS  rV;  2)  The  ORD  is  modified  only  as  a  result  of  changes  in  the  MNS;  3)  IPS,  at 
MS  IV,  describes  the  need  to  pursue  proposals  for  major  upgrades  or  modifications  to 
systems  tluu  are  still  in  production  and  to  document  the  ability  of  the  upgrade  to  satisfy 
the  need;  4)  The  TEMP  is  updated  at  MS  IV  reflecting  significant  program  or  test  plan 
changes.  The  SMM  and  RCM  are  updated  along  with  results  of  testing  completed;  5) 
The  COEA  is  updated  with  an  analysis  reflecting  costs  and  consequences  of  all 
alternatives. 


rx 
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7.  EW  SYSTEMS  T&E  DURING  PHASE  IV  -  OPERATIONS  AND  SUPPORT 
(O&S) 


7.0.1  Phase  Description.  The  O&S  Phase  overlaps  the  Production  and 
Deployment  I%ase  and  begins  at  the  time  of  Initial  Operational  Capability  (lOQ  and 
continues  until  the  EW  system  is  retired. 

7.0.2  TAP  in  The  objective  of  T&E  in  this  Phase  is  to  ensure  that  tlu  EW 
system  continues  to  meet  the  user's  mission  requiremrats  and  m  identify  any 
deficiencies  or  modifications  needed  to  inqprove  the  performance  of  the  tystem.  The 
O&S  Phase  may  require  testing  to  confirm  system  performance  and  operational 
effectiveness,  and  to  rq>roduce  problems  that  may  require  re-examination. 

7.0.2. 1  Developmental  Test  and  Evaluation.  DT&E  conducted  within  this 
phase,  if  necessary,  is  performed  to  identify  engiiKering  design  or  technical 
shortcomings  or  deficiencies  that  need  to  be  corrected. 

7.0.2.2  Operational  Test  and  Evaluation.  OT&E  is  conducted  to  ensure 
successful  attainment  of  user  operational  needs,  to  develop  and  evaluate  tactics,  and  to 
verify  system  performance  after  software  chaises,  or  the  application  of  ECPs,  or 
changes  in  the  threat.  FOT&E  may  be  conducted  to  confirm  the  performance 
improvement  and  evaluate  the  system's  operatiomd  effectiveness. 

7.0.3  Application  of  T&F.  Process  in  O&S.  The  EW  System  T&E  process  is  used 
to  plan,  conduct,  and  report  T&E  results  to  answer  tHe  decision  makers.  The  main 
objective  of  the  Process  in  this  phase  is  to  determine  whether  a  system  needs  to  revert 
to  an  earlier  Acquisition  Phase.  If  only  an  upgrade  is  required,  then  the  upgrade  is 
niade  on  the  system  and,  once  again,  expect^  outcomes  are  obtained  using  the  most 
current  models  and  simulation  systems,  the  test  scenario  is  structured,  and  testing 
occurs.  Depending  on  the  extensiveness  of  the  modification  or  change,  all  test  types 
previously  discussed  herein  are  considered. 

7.0.4  Reviews  are  held  as  required.  As  modifications  are  recommended  by  the 
user  (based  on  operational  results  and/or  field  use),  the  PM/PEO  oversees  the  T&E 
community's  panicipation  in  meetings  and  reviews  to  ensure  that  all  involved 
community  member  agencies  are  adequately  represented  and  that  noted  deficiencies  and 
recommended  improvements  are  corrected  or  modified  accordingly. 

7.0.5  Acquisition  Program  Baseline  fAPB).  The  APB  establishes  performance 
thresholds  and  objectives  to  reflect  the  new  EW  system  configuration.  In  addition,  any 
characteristics  that  change  as  a  result  of  the  modification(s)  are  reflected  in  the 
applicable  documentation  (ORD,  TEMP,  COEA,  SMM,  RCM,  and  IPS,  as  before). 


45 


7.1  Milestone  lV>Ma^or  Modification  Apinroyal.  Tlie  MS  IV  decision  detennines 
if  a  major  modification  of  the  system  is  warranted.  If  so,  the  DoD  Acquisition  Phase  to 
reenter  is  designated  in  the  ADM. 

7.2  Fluu»  rV-Operations  and  Support. 

7.2.1  This  phase  identifies  if  the  fielded  EW  system  is  meeting  the  users 
needs  and  if  any  shortcomings  and  deficiencies  need  to  be  corrected  to  inq)rove 
performance. 

7.2.2  As  a  minimum,  T&E  in  the  O&S  Phase  addresses  the  (generic) 
questions  as  depicted 
in  Figure  15  and, 
more  specifically,  as 
listed  below  as  they 
might  relate  to  each 
EW  system: 

(1) 

For  DT&E,  what  are 
the  deployed  system's 
capabilities  and 
limitations?  What  are 
the  capabilities  and 
limitations  of 
modifications  and 
upgrades? 


PtMw  IV'-OpcnKoas  A  Support 

For  the  letected  sytion. 

(1)  For  DTAE.  what  ore  ihe  depinyed  oyoten's  dauuuottotBrt  copobaiiiet  and  limiiaiiona? 
What  are  the  eapabilitica  and  tiniiiattonf  of  modifiaoioM  aod  iiptneka? 

(2)  For  OTAE.  doea  the  aynem  cratoaie  to  be  opetattonally  ntiuMe  and  effective  in 
operationai  uae?  f)o  pitipMed  madificatioai  and  npcradea  bicicart  the  auitability  4nd 
eHteiiveaeaa  of  the  ayauf 

(3}  Have  petfonnanee  otjectivea/ihreaftolda  vewua  advanced  direaia  been  validated?  Ooea 
the  ayateni  meet  iheae  periotmaiicB  objtscrivea/ihicabolda? 

(4)  b  the  TEMP  cuneni? 

(3)  Should  a  Mileaume  IV  be  directed  after  Milcaiooe  m.  what  are  the  Phaae  ID  TftE  *eiii 
citeria?* 


Rgure  15.  Phase  IV  T&E  Questions 


(2)  For  OT&E,  does  the  system  continue  to  be  operationally  suitable 
and  effective  in  operational  use?  Do  proposed  modifications  and  upgrades  increase  the 
suitability  and  effectiveness  of  the  system? 

(3)  Have  performance  objectives  and  thresholds  versus  advanced  threats 
been  validated?  Does  the  system  meet  these  performance  objectives  and  thresholds? 

(4)  Is  the  TEMP  current? 

(5)  Should  a  new  Milestone  0  be  directed  at  this  point  and  what  are  the 
iMW  Phase  0  T&E  "exit  criteria"? 
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73  EW  T&E  Process  AppUed  to  ModificationsAJpgrades. 


7.3;  1  Tntroduction.  This  section  describes  program  modifications  and/or 
upgrades  and  how  the  EW  System  T&E  Process  may  be  applied  to  each  type 
modification  and/or  upgrade  (as  defined  in  DoDI  5000.2)  of  the  existing  EW  System. 

7.3.2  Mndificariong/Maior  Modificfltinn^/TTpyniH#^ 

7.3.2.1  A  mndifieatinn  refers  to  changes  to  a  system  (whethn  for  safety,  to 
correct  a  deficiency,  or  to  improve  program  performance)  that  is  still  being  produced. 

7.3.2.2  A  fn^or  modification  is  defined  as  a  modification  that  meets  the 
criteria  of  acquisition  category  (ACAT)  I  or  n  or  is  designated  as  such  by  tl»  milestone 
reviewing  aufeority.  A  major  modification  is  necessitated  by:  1)  a  change  in  the  threat 
or  Defense  Planning  Guidance,  2)  a  deficiency  identified  during  follow-on  operational 
testing,  or  3)  operational  training  and  support,  or  an  opportunity  to  reduce  the  cost  of 
system  ownership.  Within  the  context  of  this  EW  T&E  process,  the  need  for  a  major 
modification  would  dictate  that  the  system  reenter  the  acquisition  process  at  Phase  0 
and  restart  with  the  definition  of  requirements  tc  begin  a  new  conceptual  design. 

7.3.2.3  An  upgrade  is  a  change  to  a  system  (w)»tfaer  for  safety,  to  correct  a 
deficiency,  or  to  inqirove  program  performance)  to  a  system  that  is  out  of  production. 
Upgrades  are  part  of  the  Milestone  0  decision  process. 

7.3.3  T&E  in  Modifications  or  Upgrades.  Test  and  Evaluation  associated  with 
modifications  or  upgrades  is  consistent  with  the  previously  discussed  purpose  for  T&E, 
that  is,  to  provide  T&E  information  for  decision  makers.  In  keeping  with  this  concept, 
T&E  is  performed:  1)  to  ensure  that  the  fielded  EW  system  continues  to  provide  the 
capabilities  required  to  meet  the  mission  need,  and  2)  to  identify  shortcomings  or 
deficiencies  that  must  be  corrected  to  improve  performance. 

OT&E  is  also  conducted  to  develop  and  evaluate  tactics  and  verify  system 
performaiue  after  software  changes.  The  OTA  may  be  required  to  conduct  testing 
depending  on  the  stage  of  development  and  the  risks  involved.  The  OT&E  objectives 
for  an  EW  system  major  modification  are  essentially  the  same  as  for  an  EW  system  on 
its  first  pass  through  the  acquisition  process.  An  OT&E  is  usually  conducted  to  support 
a  production  decision  for  the  modification. 

7.3.4  EW  System  T&E  Process  Application.  The  DoD  EW  System  T&E 
Process  is  easily  tailored  to  fit  modifications  or  upgrades.  Depending  on  the  type  of 
modification,  whether  a  major  modification  involving  a  system  reentering  Phase  0,  or 
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an  upgrade  wherdiy  the  system  sinqjiy  requires  additional  testing  to  prove  its  ability  to 
meet  mission  capabilities,  the  process  is  sufficiently  flexible  to  accommodme  the 
system's  needs.  The  T&E  process  allows  the  PM  to  modify,  as  necessary,  the  amount 
and  type  of  testing  necessary  to  obtain  required  critical  dam  on  the  system.  If  the 
system  has  met  accq)tabie  levels  of  performance  at  the  appropriate  phase  of  testing, 
then  it  will  proceed  to  the  next  phase  of  testing.  If  the  system  has  ^ed  to  achieve  the 
established  perfotinance  criteria,  then  the  system  will  be  reexamined  from  the 
standpoint  of:  1)  the  appropriateness  of  the  established  performance  critoria,  2)  the 
accuracy  to  which  the  performance  was  estimated,  3)  the  ability  of  tibe  system  to 
achieve  the  mission  need,  and  4)  the  cost  efifecdveness  of  die  modification  or  upgrade. 

The  MDA  makes  a  decision  based  upon  the  ability  of  the  sysmm  to  achieve 
critical  performance  parameters  and  to  proc^  to  the  next  step  within  the  acquisition 
process,  or  the  MDA  may  terminate  the  modification  program  based  upon  one  of  the 
reasons  above. 

7.3.5  Reviews  are  held  as  directed  by  the  MDA.  If  a  COEA  is  required,  the 
MDA  specifies  the  elements  requiring  further  analysis.  As  part  of  the  program  review, 
alternatives  to  the  modification  must  be  considered.  This  confirms  that  all  performance 
objectives  have  been  met  and  any  new  threat  assessment  has  been  validated. 

7.3.6  Documentation  is  updated  in  accordance  with  DoD  5000  series  for  any 
system  undergone  modification  or  upgrade.  The  COEA  is  updated  to  ensure  that  the 
system  still  presents  a  cost  effective  solution  in  meeting  the  needs  of  the  user.  The 
TEMP  is  updated  to  ensure  that  the  proposed  modifications  or  upgrades  occur  in 
logical  progression  and  that  there  is  an  integrated  approach  to  testing  these 
modifications.  And,  finally,  the  STAR  is  updated  to  ensure  that  the  modified  system 
still  has  the  potential  of  engaging  and  defeating  current,  validated  threats  and  that 
future  modifications  take  such  threats  into  consideration. 
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8.  SUMMARY 


Tl»  DoD  T&E  Process  for  Electronic  Warfare  Systems  is  a  disciplined  and 
consistent  process  designed  to  provide  T&E  information  to  the  decision  makers.  This 
information,  when  applied  with  other  program  information,  enables  the  appropriate 
decision  mairing  elements  to  arrive  at  concrete  decisions  regarding  system  maturation. 

This  Process,  ctmqnised  of  five  steps,  is  iterative  and  assesses  military  worth  in 
terms  of  the  system  contribution  in  meeting  military  iteeds.  The  T&E  mMhodology 
provides  for  consistent  testing  of  EW  systems,  provides  for  conaprehensive  T&E  of 
performance  objectives  and  thresholds  using  styhisticated  testing  resources,  results  in 
an  accurate  documentation  of  test  result  data,  ai^  calls  for  timely  analysis  to  be 
forwarded  to  decision  makers  in  the  form  of  DT&E  and  OT&E  assessments.  Consistent 
and  accurate  information  provided  to  decision  makers  is  necessary  to  achieve  timely 
system  development  which  satisfies  user  needs. 

This  T&E  Process  may  be  applied  to  the  acquisition  of  an  EW  system  in  all  DoD 
Components.  It  is  consistent  with  the  DoD  Acquisition  Process  and  appropriately 
defined  within  DoD  5000  series  documentation.  It  is  expressly  intended  to  assist 
acquisition  executives,  program  managers,  program  test  coordinators,  test  organization 
personnel,  and  others  involved  in  the  management  of  T&E  of  new,  or  highly  modified, 
EW  systems. 
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APPENDIX  A 


ABBREVIATIONS  AND  ACRONYMS 


AARM 

Anti-Air  Anti-radiadon  Miasiie 

AASPEM 

Advanced  Air-to-air  System  Performance  Model 

ACAT 

Acquisition  Cat^ory 

ADM 

Acquisition  Decision  Memorandum 

ALARM 

Advanced  Low  Althude  Radar  Modd 

APB 

Acquisition  Program  Baseline 

ATD 

Advanced  Technology  Demonstration 

CE&D 

Concept  Exploration  &  Definition 

acs 

Chairman  of  the  Joint  Chie6  of  Staff 

COEA 

Cost  and  Operational  Effectiveness  Analysis 

COI 

Critical  Operational  Issue 

CT? 

Critical  Technical  IHuameter 

C2W 

Command  &  Control  Warfare 

DECM 

Deceptive  Electronic  Countermeasures 

DEMATAL 

Demonstration  &  Validation  (Phase) 

DIA 

Defense  Intelligence  Agency 

DoD 

Department  of  Defense 

DoDD 

DoD  Directive 

DoDI 

DoD  Instruction 

DSM 

Digital  System  Model 

A-1 


DSMC 

DT&E 

EA 

ECCM 

ECM 

ECP 

EMD 

EOA 

EP 

ERP 

ES 

EW 

FOT&E 

HARM 

HTTL 

IOC 

IPS 

IR 

IRCM 

JCS 

JC2WC 

JEWC 

J-MASS 


IX^eose  Systems  Managemeot  College 
Develqmieni  Test  and  Evahiation 
EtectroQic  Attuk 

Etectronic  Counter  Countenneasures 

Electronic  Countermeasuies 

Engmeering  Change  Prcqmsai 

Engineering  and  Manufimturing  Development 

Early  (^jerational  Assessment 

Electronic  Protection 

Effective  Radiated  Power 

Electronic  Support 

Electronic  Warfue 

Follow<on  Operational  T&E 

High-speed  Anti-Radiation  Missile 

Hardware-in-the-ioop 

Initial  Operational  Capability 

Integrated  Program  Summary 

Infra-red 

Infra-r«i  Countenneasures 
Joint  Chiefs  of  Staff 

Joint  Command  and  Control  Warfare  Center  (formerly  JEWC) 
Joint  Electronic  Warfare  Center 
Joint  Modeling  and  Simulation  System 
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MDA 

Milestone  Deciskm  AudK)rity 

MNS 

Mission  Need  Statement 

MOE 

Measuiemait  of  Effectiveness 

MOP 

Measure  of  Performance,  Menaorandum  of  Policy 

MS 

Milestone 

MTI 

Moving  Target  Indicator 

OPSEC 

Operational  Security 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OTA 

Operational  Test  Agency 

OT&E 

Operational  T&E 

PDA 

Program  Decision  Authority 

PEO 

Program  Executive  Officer 

PM 

Program  Manager 

PP 

Production  Prototype 

PSYOP 

Psychological  Operations 

P3I 

Preplanned  Produa  Improvement 

RAM 

Reliability,  Availability,  and  Maintainability 

RCM 

Requirements  Correlation  Matrix 

RCS 

Radar  Cross  Section 

RDA 

Research,  Development,  and  Acquisition 

SAM 

Surface-to-Air  Missile 
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SEAD 


Siqnnessioa  of  Eoeii^  Air  Defenses 
SIL  i^rstem  Int^ratkm  Lsbontuy 

SMM  System  Maturity  Matrix 

STAR  System  Threat  Assessment  Report 

SUT  ^stem  Under  Test 

TEMP  T&E  Master  Plan 

TST  Tactical  SIGINT  Technology 

USD(A&T)  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
WARM  Wartime  Reserve  Modes 
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DEFINmONS 


ACQUISITION  CATEGORY  (ACAT)  Categraies  established  acquisitioii 
pn^nuns  to  facilitate  decemialized  decision  making  and  ezecutkm  and  compliance  with 
^atutcMrily  inqx)8ed  requiiemeots.  The  cat^ories  *<****""««"  the  level  of  review, 
decision  autlioii^,  and  applicabte  procedures.  (DoDI  S000.2) 

a.  ACAT  I.  Has  been  designated  by  dm  Under  Secretary  of  Defense 
(Acquisidoa  and  Technology)  as  an  ACAT  I  program  or  has  an  eventual  RDT&E 
expmiditure  of  more  than  $200  million  and  a  {nocurement  expenditure  of  more  than  $i 
billion  in  1980  constant  dollars. 

b.  ACATn.  Has  been  designated  by  the  DoD  bead  component  as  an  AC  AT  n 
program.  Has  an  eventual  RD&E  expenditure  of  name  than  $75  i^ion  and  a 
procurement  expenditure  of  $300  MfiUion  in  1980  constant  dollars. 

c.  ACATm.  Programs  not  meetup  the  criteria  for  AC  AT  I  or  n  that  have 
been  designated  Category  II  by  the  DoD  Component  Acquisition  Executive. 

d.  ACAT  rv.  Other  acquisition  programs  for  which  the  milestone  decision 
authority  should  be  delegated  to  a  level  below  that  required  for  ACAT  m. 

ACQUISITION  PROCESS  The  basis  for  comprehensive  naanagement  and  the 
progressive  decision  making  associated  with  program  maturation.  The  acquisition 
process  consists  of  five  major  milestone  decision  points  and  five  phases  (Concept 
Exploration  and  Definition,  Demonstration  and  Validation,  Engineering  and 
Manufecturing  Develc^nnent,  Production  and  Deployment,  and  Operations  and 
Support).  (DoDI  5000.2) 

ACQUISITION  DECISION  MEMORANDUM  (ADM)  A  memorandum  signed  by 
the  milestone  decision  authority  that  documents  decisions  made  and  the  exit  criteria 
established  as  the  result  of  a  milestone  decision  review  or  in-process  review.  (DoDI 
5000.2) 

ACQUISITION  PLAN  A  formal  written  document  reflecting  the  specific  actions 
necessary  to  execute  tte  approach  established  in  the  appmved  acquisition  strategy  and 
guidii^  contractual  implementation.  (FAR  Subpart  7.1 ) 
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ACQUISITION  PROGRAM  A  directed,  funded  effort  that  is  designed  to  provide  a 
new  or  improved  materiel  capability  in  response  to  a  valklated  need.  (DoDD  SOOO.l) 

ACQUISITION  mOGRAM  BASELINE  (APB)  A  document  requited  for  all 
acquisition  categories  that  entixxiies  the  cost,  schedule,  and  performance  objectives  for 
the  program.  (DoDI  AXX).2) 

ACQUISITION  STRATEGY  The  master  plan  for  program  execution  from  program 
initiation  throu^  post-production  siqiport.  It  is  to  be  developed  in  sufiRciem  detail  to 
establish  the  managerial  approach  that  will  be  used  to  direa  and  control  all  elements  of 
the  acquisition  to  achieve  program  objectives.  (DoDI  5000.2) 

ANTI-RADIATION  MISSILE  A  missile  which  homes  passively  on  a  radiation 
source.  (CJCS  MOP  6) 

AVAILABILITY  A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
committafale  state  at  the  start  of  a  mission  when  the  mission  is  called  for  at  an  unknown 
(random)  time.  (DoDI  5000.2) 

AVIONICS  Electrical  and  electronic  systems  and  devices  used  in  aviation,  missilery, 
and  astronautics.  (Commonly  used  military  jargon  that  is  a  contraction  of  aviation 
electronics)  (EW  T&E  Task  Force) 

BRASSBOARD  CONFIGURATION  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  and  operational  data.  It  normally 
is  a  model  sufficientiy  hardened  for  use  outside  of  laboratory  environments  to 
demonstrate  the  technical  and  operational  principles  of  immediate  interest.  It  may 
resemble  the  end  item,  but  is  not  intended  for  use  as  the  end  item.  (EW  T&E  Task 
Force) 

BREADBOARD  CONFIGURATION  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  data.  It  normally  is  configured 
only  for  laboratory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  It 
may  ntH  resonble  the  end  item  and  is  not  inteiuied  for  use  as  the  projected  end  item. 
(EW  T&E  Task  Force) 

CLOSED  LOOP  A  term  used  frequently  in  testing  electronic  systems  to  characterize 
a  situation  in  which  both  the  SUT  and  a  stimulating  system  (usually  a  threat  system) 
can  mutually  respond  and  interact,  and  the  responses  and/or  behavior  of  both  are 
measured  or  recorded.  (EW  T&E  Task  Force) 

COMMAND  AND  CONTROL  (C^)  The  exercise  of  authority  and  direction  by  a 
properly  designated  commander  over  assigned  or  attached  forces  in  the  accomplishment 
of  tlK  mission.  C^  functions  are  performed  through  an  arrangement  of  personnel. 
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eqaipment,  ccnmnumcatioiis,  c(»ii|)uters,  fiidiides,  and  procedures  enq)l(^ed  by  a 
commander  in  planning,  directing,  coordinating,  and  controilii^  forces  and  operations 
in  accon^lishinent  of  the  missirm.  (Joint  Pub  1-02) 

COMMONALITY  A  quality  udiich  applies  to  materiel  or  systems:  a.  possessing  like 
and  interchangeable  charactmistics  enabling  each  to  be  utilized,  or  operated  and 
maintained,  by  persormel  trained  on  the  odms  without  additional  specialized  training; 
b.  having  interchaiqieable  parts  and/or  cotrqwnents;  or  c.  qjplying  to  consumable  items 
interchai^eably  equivalent  widiout  adjustmeoL  (Joint  Pub  1-02) 

COMPATIBILITY  The  capabili^  of  two  or  more  items  or  cotqponents  of  equqnnent 
or  material  to  exist  or  function  in  the  same  system  or  environment  without  mutual 
interference.  (Joint  Pub  1-02) 

CONCEPT  A  notion  or  statement  of  an  idea,  expressing  how  something  might  be 
done  or  accomplished,  that  may  lead  to  an  accepted  procedure.  (Joint  Pub  1-02) 

CONCEPT  BASELINE  The  baseline  approved  at  Milestone  I  that  contains  broad 
objectives  and  thresholds  for  key  cost,  schedule,  and  performance  parameters.  It  is 
submitted  by  the  designated  component  official  throu^  the  MDA  as  a  stand-alone  part 
of  the  MS  I  documentation.  (DoDI  5(XX).2) 

COST  AND  OPERATIONAL  EFFECTIVENESS  ANALYSIS  (COEA)  An  analysis 
of  the  estimated  costs  and  operational  effectiveness  of  alternative  materiel  systems  to 
meet  a  mission  need  and  the  associated  program  for  acquiring  each  alternative.  (DoDI 
5000.2) 

COST  EFFECTIVENESS  A  measure  of  the  operational  capability  added  by  a  system 
as  a  fenction  of  its  life-cycle  cost.  (DoDI  5000.2) 

CRITICAL  DESIGN  REVIEW  (CDR)  A  review  conducted  to  determine  that  the 
detailed  design  satisfies  the  performance  and  et^ineering  requirements  of  the 
development  specification;  to  establish  the  detailed  design  compatibility  among  the  item 
and  other  items  of  equipment,  fecilities,  computer  programs,  a^  personnel;  to  assess 
producibility  and  risk  areas;  and  to  review  tte  preliminary  product  specifications. 
Conducted  during  Phase  I,  DEM/VAL  (for  prototypes)  aixl  Phase  II,  EMD.  (DoDI 
5000.2) 

CRITICAL  OPERATIONAL  ISSUE  (COD  A  key  operational  effectiveness  or 
operational  suitability  issue  that  must  be  examined  in  operational  test  and  evaluation  to 
determine  the  system's  capability  to  perform  its  mission.  A  critical  operational  issue  is 
normally  phrased  as  a  question  to  be  answered  in  evaluating  a  system's  operational 
effectiveness  and/or  operational  suitability.  (DoDI  5000.2) 
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CRITICAL  SYSTEM  CHARACTERISTICS  Design  features  tlutt  determine  how 
well  the  proposed  concept  or  system  wiU  function  in  its  intended  operational 
environment.  They  include  survivability;  transportabiliQ';  energy  efficiency;  and 
interoperability,  standardization,  and  conq)atibility  with  other  forces  and  systems 
includii^  su{^tt  infrastructure.  (DoDl  5000.2) 

CRITICAL  TECHNICAL  PARAMETERS  Design-specific  system  performance 
and/or  mchnicai  parameters  characterizing  system  capabilities  required  to  achieve  key 
performance  threshold  values.  (DoDI  5000.2) 

DESIGN  The  process  of  converting  specifications  into  a  visual  or  mathematical 
representation  of  the  system.  (EW  T&E  Task  Force) 

DETECTION  The  perception  of  an  object  of  possible  military  interest  but 
unconfirmed  by  recognition.  (Joint  Pub  1-02) 

DEVELOPMENT  BASELINE  Baseline  confining  more  detailed  and  refined 
objectives  and  thresholds  for  key  cost,  schedules  and  performance  parameters  then  the 
concept  baseline.  It  is  submitted  as  a  stand-alone  part  of  the  Milestone  n 
documentation.  (DoDI  5000.2) 

DEVELOPMENT  TEST  AND  EVALUATION  (DT&E)  Activity  that  is  conducted 
to  demonstrate  that  the  engineering  design  and  development  process  is  complete.  It  is 
used  to  reduce  risk,  validate  and  qualify  system  design  and  to  ensure  that  tite  end- 
product  is  ready  for  operational  test  and  evaluation.  The  DT&E  is  the  responsibility  of 
the  materiel  developer.  (DSMQ 

DIRECTED  ENERGY  An  umbrella  term  covering  technologies  that  relate  to  the 
production  of  a  beam  of  concentrated  electromagnetic  energy  or  atomic  or  subatomic 
particles.  Also  called  DE.  (CJCS  MOP  6) 

DIRECTED-ENERGY  WEAPON  A  system  using  directed-energy  primarily  as  a 
direct  means  to  damage  or  destroy  enemy  equipment,  facilities,  and  personnel.  (CJCS 
MOP  6) 

DoD  COMPONENT  ACQUISITION  EXECUTIVE  A  single  official  within  a  DoD 
Component  who  is  responsible  for  all  acquisition  functions  within  that  Component. 
This  includes  Service  Acquisition  Executives  for  the  Military  Departments  and 
Acquisition  Executives  in  other  DoD  Components  who  have  acquisition  management 
responsibilities.  (DoDI  5000.2) 

EARLY  OPERATIONAL  ASSESSMENT  An  operational  assessment  conducted 
prior  to,  or  in  support  of.  Milestone  II.  (DoD  5(X)0.2) 
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ELECTROMAGNETIC  COMPATIBILnY  (EMC)  The  ability  of  systems 
equ^nnent,  and  devices  that  utilize  the  electromagnetic  spectrum  to  operate  in  their 
intended  operational  environments  without  suffering  unacceptable  degradation  or 
causing  unintentional  degradation  because  of  electromagnetic  radiation  or  response.  It 
involves  the  application  of  sound  electromagnetic  spectrum  management;  system, 
equipment,  aM  device  design  configuration  that  ensures  interference-free  operation; 
and  clear  concepts  and  doctrines  that  Ttmrimiye  operational  effectiveness.  (CJCS  MOP 
6) 

ELECTROMAGNETIC  DECEPTION  The  delil>eiatB  radiation,  re-radiation, 
alteration,  suppression,  absorption,  denial,  enhancement,  or  reflection  of 
electromagnetic  energy  in  a  manner  intended  to  convey  misleading  information  to  an 
eittmy  or  to  enemy  electromagnetic-dependent  weapons  thereby  degrading  or 
neutralizing  the  enemy's  combat  capability.  (CJCS  MOP  6> 

ELECTROMAGNETIC  ENVIRONMENT  (EME)  The  resulting  product  of  the 
power  and  time  distribution,  in  various  frequency  ranges,  of  the  radiated  or  conducted 
electromagnetic  emission  levels  that  may  be  encountered  by  a  military  force,  system,  or 
platform  when  performii^  its  assigned  mission  in  its  intended  operational  environment. 
(Joint  Pub  1-02) 

E^CTROMAGNETIC  HARDENING  Action  taten  to  protect  personnel,  facilities, 
ai^or  equipment  by  filtering,  attenuating,  grounding,  bonding,  and/or  shielding 
against  tmdrairable  effects  of  electromagnetic  energy.  (CJCS  MOP  6) 

ELECTROMAGNETIC  INTERFERENCE  (EMI)  Any  electromagnetic  disturbaiKe 
that  interrupts,  obstructs,  or  otherwise  degrades  or  limits  the  effective  performance  of 
electronics/electrical  equipment.  It  can  be  induced  intentionally,  as  in  some  forms  of 
electronic  warfare,  or  unintentionally,  as  a  result  of  spurious  emissions  and  responses, 
intermodulation  products,  and  the  like.  (CJCS  MOP  6) 

ELECTROMAGNETIC  JAMMING  The  deliberate  radiation,  re-radiation,  or 
reflection  of  electromagnetic  energy  for  the  purpose  of  preventing  or  reducing  an 
enemy’s  effective  use  of  the  electromagnetic  spectrum,  and  with  the  intent  of  degrading 
or  neutralizing  the  enemy’s  combat  capability.  (CJCS  MOP  6) 

ELECTRONIC  COUNTER-COUNTERMEASURES  (ECCM)  The  division  of 
electronic  warfare  involving  actions  taken  to  ensure  friendly  effective  use  of  the 
electromagnetic  spectrum  despite  the  enemy's  use  of  electronic  warfare.  (Joint  Pub  1- 
02) 

ELECTRONIC  COUNTERMEASURES  (ECM)  The  division  of  electronic  warfare 
involving  action  taken  to  prevent  or  reduce  the  enemy’s  effective  use  of  the 
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eiectFomagnetic  spectrum.  It  includes  electronic  jamming  and  electronic  deception. 
(Joint  Pub  1-02) 

ELECTRONIC  MASKING  Ibe  controlled  radiation  of  electromagnetic  energy  on 
friendly  frequencies  in  a  manner  to  protect  tfae  emissions  of  friendly  communications 
and  electronic  systems  against  enemy  ESM/SIGINT  (electronic  warfare  s(^}po]t 
measures/signais  inmlligence),  without  significantly  degrading  the  operation  of  friendly 
systems.  (OCSMOPd) 


ELECTRONIC  WARFARE  EXPENDABLES  Nonrecoverable  EW  devices  such  as 
chaff,  flares,  unmanned  vdiicles,  decoys,  and  unattended  jammers.  (CJCS  MOP  6) 


EXECTRONICS  INTELLIGENCE  (ELINT)  Technical  and  intelligence  information 
derived  from  foreign  non-communications  electroim^netic  radiations  emanating  from 
other  than  nuclear  detonations  or  radioactive  sources.  (CJCS  MOP  6) 


EMISSION  CONTROL  (EMCON)  The  selective  and  controlled  use  of 
electrom^netic,  acoustic,  or  other  emitters  to  optimize  command  and  control 
capabilities  while  minimizing,  for  operations  security  (OPSEC),  detection  by  enemy 
sensors;  to  minimize  mutual  interference  among  friendly  systems;  and/or  to  execute  a 
military  deception  plan.  (CJCS  MOP  6) 

ENGINEERING  CHANGE  PROPOSAL  (ECP)  A  formal  document  used  to  make 
engineering  changes  in  an  existing  contract.  (DSMC) 

ENGINEERING  PROTOTYPE  (EP)  A  development  model  of  a  unit  that  is  close  to 
production.  The  term  may  apply  to  circuitry,  a  device  (black  box)  or  a  system,  and 
may  be  in  a  breadboard  (technical)  configurmion.  (EW  T&E  Task  Force) 

EVALUATION  The  technical  and/or  operational  smdy  and  investigations  by  a 
developing  and/or  operational  agency  to  determine  the  technical  and/or  operational 
suitability  and  effectiveness  of  material,  equipment,  or  a  system  for  use  in  the  military 
Services.  (Joint  Pub  1-02) 

EVALUATION  CRITERIA  Standards  by  which  accomplishments  of  required 
technical  and  operational  effectiveness  and/or  suitability  characteristics  or  resolution  of 
operational  issues  may  be  assessed.  (DoD  5000.2) 

EXIT  CRITERIA  Program  specific  accomplishments  that  must  be  satisfactorily 
demonstrated  before  an  effon  or  program  can  proceed  further  in  the  current  acquisition 
phase  or  transition  to  the  next  acquisition  phase.  Exit  criteria  may  include  such  factors 
as  critical  test  issues,  the  attainment  of  projected  growth  curves  and  baseline 
parameters,  and  the  decision  to  proceed  further.  Exit  criteria  are  specific  to  each 
acquisition  phase.  (DoD  5000.2) 
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FOLLOW  ON  OPERATIONAL  TEST  AND  EVALUATION  Test  and  evaluation 
that  is  necessary  durii^  and  after  the  production  period  to  refine  the  estimates  made 
during  operational  test  and  evaluation,  to  evaluate  changes,  and  to  reevaluate  the 
system  to  ensure  it  continues  to  meet  operational  needs  and  retains  its  effectiveness  in  a 
new  environment  or  against  a  new  threat  (DoO  5000.2) 

FULL  RATE  PRODUCTION  Production  of  economic  quantities  following 
stabilization  of  the  system  design  and  prove-out  of  the  production  process.  (DoD 
5000.2) 

IDENTIFICATION  The  process  of  determining  the  friendly  or  hostile  character  of  an 
unknown  detected  contact.  (Joint  Pub  1-02) 

IMITATIVE  ELECTROMAGNETIC  DECEPTION  The  introduction  of 
electromagnetic  energy  into  enemy  systems  that  imitates  enemy  emissions.  (CJCS 
MOP  6) 

INITIAL  OPERATIONAL  CAPABILITY  The  first  attainment  of  the  capability  to 
employ  effectively  a  weapon,  iton  of  equipment,  or  system  of  approved  specific 
ch^cteristics,  and  which  is  manned  or  operated  by  a  trained,  equipped,  and  supported 
military  unit  or  force.  (DoDI  5000.2) 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E)  All  operational 
test  and  evaluation  conducted  on  production  or  production  representative  articles,  to 
support  the  decision  to  proceed  beyond  low-rate  initial  production.  It  is  conducted  to 
provide  a  valid  estimate  of  expected  system  operational  effectiveness  and  operational 
suitability.  (DoDI  5000.2) 

INSTRUMENTATION  The  installation  and  use  of  electronic,  gyroscopic,  and  other 
instruments  for  the  purpose  of  detecting,  measuring,  recording,  telemetering, 
processing,  or  analyzing  different  values  or  quantities  as  encountered  in  the  flight  of  an 
aircraft,  missile,  or  spacecraft.  Instrumentation  applies  to  flight-bome,  sea-borne,  and 
ground-based  equipment.  (AFM  11-1) 

INTEGRATED  PROGRAM  SUMMARY  (IPS)  A  DoD  component  document 
prepared  and  submitted  to  the  milestone  decision  authority  in  support  of  Milestone  I-IV 
reviews.  It  succinctly  highlights  the  stams  of  the  program  and  its  readiness  to  proceed 
into  the  next  phase  of  the  acquisition  process.  (DoDI  5000.2) 

INTEROPERABILITY  The  ability  of  systems,  units,  or  forces  to  provide  services  to 
or  accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so 
exchanged  to  operate  effectively  together.  (DoDI  5Q00.2) 
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JOINT  REQUIREMENTS  OVERSIGHT  COUNCIL  (JROQ  The  JROC  is 
responsible  to  the  Chairman  of  the  Joint  Chiefs  of  Staff  for  assessing  military 
requiren^nts  in  support  of  the  defense  acquisition  process.  The  Vice  Chairman  of  ti»; 
Joint  Chiefs  of  Staff  chairs  the  Council  and  decides  all  matters  before  the  Council.  The 
permanent  members  include  the  Vice  Chiefs  of  the  Army  and  Air  Force,  the  Vice  Chief 
of  Naval  Operations,  and  the  Assistant  Commandant  of  the  Marine  Corps.  The  Council 
directly  support  the  Defense  Acquisition  Board  through  the  review,  validation,  and 
approval  of  military  requirements  at  the  stan  of  the  acquisition  process,  prior  to  each 
roestone  review,  or  as  requested  by  the  Under  Secietaxy  of  Defense  for  Acquisition 
and  T«;hnology.  (DoDI  5(XX).2) 

KEY  PARAMETERS  Those  parameters  that,  if  the  thresholds  are  not  met,  the 
milestone  decision  authority  would  require  a  reevaluation  of  alternative  concepts  or 
design  approaches.  They  are  derived  from  the  ORD  and  included  as  thresholds  in 
baseline  documentation.  (DoDI  5000.2) 

LIFE-CYCLE  COST  The  total  cost  to  the  Government  of  acquisition  and  ownership 
of  that  system  over  its  useful  life.  It  includes  the  cost  of  development,  acquisition, 
support  and,  where  applicable,  disposal.  (DoDI  5000.2) 

LIVE  FIRE  TEST  AND  EVALUATION  REPORT  A  report  to  be  submitted  by  the 
Secretary  of  Defense  (or  as  delegated  to  the  Under  Secretary  of  Defense  for  A&T  for 
applicable  (covered  under  the  live  fire  test  law)  ACAT I  programs  or  the  Director, 
Defense  Research  &  Engineering,  for  applicable  ACAT  H  programs)  to  the 
Committees  on  Armed  Services  and  on  Appropriations  of  the  Senate  and  the  House  of 
Representatives  prior  to  a  decision  to  proceed  beyond  low-rate  initial  production 
(LRIP).  It  is  also  required  for  a  covered  product  improvement  program  of  any 
acquisition  category  which  is  likely  to  significantly  affect  the  survivability  of  a  covered 
major  system  or  the  lethality  of  a  major  munition  or  missile  produced  under  an  ACAT 
I  or  II  program.  (DoDI  5000.2) 

LIVE  SIMULATION  A  component  of  simulation  that  involves  operations  with  real 
force  or  personnel  and  real  equipment  or  test  items  in  the  air,  on  the  ground,  on  or 
below  the  sea,  or  in  a  test  facility.  (Defense  Science  Board) 

LOGISTICS  SUPPORT  ABILITY  The  degree  to  which  planned  logistics  support 
(including  test,  measurement,  and  diagnostic  equipment;  spares  and  repair  parts: 
technical  data;  support  facilities;  transportation  requirements;  training;  manpower;  and 
software  support)  allow  meeting  system  availability  and  wartime  usage  requirements. 
(DoDI  5000.2) 

LOW  RATE  INITIAL  PRODUCTION  (LRIP)  The  production  of  a  system  in 
limited  quantity  to  provide  articles  for  OT&E.  to  establish  an  initial  production  base. 
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and  to  permit  an  orderly  increase  in  the  protbunion  rate  sufficient  to  lead  to  full-rate 
production  upon  successbil  completion  of  operational  testing.  (DoD  5000.2) 

MAlNTAlNABILiTY  Hie  ability  of  an  item  to  be  retained  in  or  restored  to  a 
specified  condition  when  maintenance  is  performed  by  personnel  having  specified  skill 
levels,  using  prescribed  procedures  and  resources,  at  each  prescribed  level  of 
maintenance  and  rqiair.  (DoDI  5000.2) 

MAJOR  MODIFICATION  A  modification  that  in  and  of  itself  meets  the  criteria  of 
acquisition  category  (AC AT)  I  or  n  or  is  designated  as  such  by  the  milestone  reviewing 
authority.  A  major  modification  is  necessitated  by:  1)  a  change  in  the  threat  or  Defense 
Plannii^  Guidance,  2)  a  deficiency  identified  during  follow-on  operational  testing,  or 
3)  operational  training  and  support,  or  an  opportunity  to  reduce  the  cost  of  system 
ownership.  Within  the  context  of  this  EW  T&E  process,  the  need  for  a  major 
modification  would  dictate  that  the  system  reenter  the  acquisition  process  at  Phase  0 
and  restart  with  the  definition  of  requirements  to  begin  a  new  conceptual  design.  (DoDI 
5000.2) 

MANIPULATIVE  ELECTROMAGNETIC  DECEPTION  Actions  to  eliminate 
revealing,  or  convey  misleading,  electromagnetic  telltale  indicators  that  may  be  used  by 
hostile  forces.  (C^CS  MOP  6) 

MEASURES  OF  EFFECTIVE^^S  (MOEs)  Tools  that  assist  in  discriminating 
among  a  number  of  alternatives,  normally  to  answer  a  Critical  Operational  Issue  (COI). 
They  show  how  the  alternatives  compare  in  meeting  functional  objectives  and  mission 
needs.  They  are  "predictions"  of  how  a  system  will  perform  when  fielded,  aiKi  are 
usually  expressed  as  a  probability  or  likelihood,  and  usually  inferred.  (DoD  5(X)0.2- 
M) 

MEASURES  OF  PERFORMANCE  (MOPs)  MOPs  are  quantitative  and  qualitative 
measures  of  a  system's  performance  or  system  characteristic.  MOPs  indicate  the 
degree  to  which  a  system  performs  the  task  or  meets  a  requirement  under  specified 
conditions.  MOPs  should  address  system  capabilities  and  characteristics  and  should 
relate  to  the  MOE(s)  such  that  the  effect  of  a  change  in  the  MOP  can  be  related  to  a 
change  in  the  MOE.  (AR  73-1) 

MEASUREMENT  AND  SIGNATURE  INTELLIGENCE  (MASINT)  Scientific  and 
technical  intelligence  information  obtained  by  quantitative  and  qualitative  analysis  of 
data  (metric,  angle,  spatial,  wavelength,  time  dependence,  modulation,  plasma,  and 
hydromagnetic)  derived  from  specific  technical  sensors  for  the  purpose  of  identifying 
any  distinctive  features  associated  with  the  source,  emitter,  or  sender  and  to  facilitate 
subsequent  identification  and/or  measurement  of  the  same.  (CJCS  MOP  6) 
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MILESTONE  DECISION  AITTHORITY  Tbe  individual  designated  in  accordance 
with  criteria  est{d}lished  by  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  to  approve  entry  of  an  acquisition  program  into  the  next  phase.  (DoDD 
5000.1) 

MILESTONES  Major  decision  points  that  separate  the  phases  of  an  acquisition 
program.  (DoDD  S0(X).l) 

MILITARY  DECEPTION  Actions  executed  to  mislead  foreign  decision  makers, 
causing  them  to  derive  and  accept  desired  tqtpredations  of  military  curabilities, 
intentions,  operations,  or  other  activities  thu  evoke  foreign  actions  that  contribute  to 
the  originator's  objective.  (CJCS  MOP  30) 

MINIMUM  ACCEPTABLE  OPERATIONAL  REQUIREMENT  The  value  for  a 
performance  or  technical  parameter  that  is  required  to  provide  a  system  capability  that 
will  satisfy  the  validated  mission  need.  Also  known  as  the  performance  threshold. 
(DoDI  5000.2) 

MISSION  NEED  STATEMENT  (MNS)  A  statemem,  expressed  in  broad  operational 
terms,  of  operational  ciq>ability  required  to  perform  an  assigned  mission  or  to  correct  a 
deHciency  in  existing  capability  to  perform  tlra  mission.  (DoDI  5000.2) 

MISSION  RELIABILITY  The  probability  that  the  system  will  perform  mission 
essential  functions  for  a  period  of  time  under  the  conditions  stated  in  the  mission 
profile.  (DoDI  5000.2) 

MODEL  A  representation  of  an  actual  or  conceptual  system  that  involves 
mathematics,  logical  expressions,  or  computer  simulations  that  can  be  used  to  predict 
how  the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of 
hostile  environments.  (DoDI  5(X)0.2) 

MODIFICATION  A  change  to  a  system  (whether  for  safety,  to  correct  a  deficiency, 
or  to  improve  program  performance)  that  is  still  being  produced.  (DoDI  50(X).2) 

NONDESTRUCTIVE  ELECTRONIC  WARFARE  Those  EW  actions,  not 
including  employment  of  WARM,  that  deny,  disrupt,  or  deceive  rather  than  damage  or 
destroy.  (CJCS  MOP  6) 

NON-DEVELOPMENT AL  ITEM  1.  Any  item  of  supply  that  is  available  in  the 
commercial  marketplace;  2.  Any  previously  developed  item  of  supply  that  is  in  use  by 
a  department  or  agency  of  the  United  States,  a  State  or  local  government,  or  a  foreign 
government  with  which  the  United  States  has  a  mumal  defense  cooperation  agreement; 
3.  Any  item  of  supply  described  in  definition  1  or  2.,  above,  that  requires  only  minor 
modification  in  order  to  meet  the  requirements  of  the  procuring  agency;  or  4.  Any  item 
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of  supply  that  is  currently  being  produced  that  does  not  meet  the  requirements  of 
definition  1.,  2.,  or  3.  atove,  solely  because  of  the  item  is  not  yet  in  use  or  is  not  yet 
available  in  the  conunercial  markeq)iace.  (DoDI  5000.2) 

OBJECTIVE  Value  beyond  the  threshold  that  could  potentially  have  measurable, 
beneficial  impact  on  capability  or  operations  and  su{^tt  above  that  provided  by  the 
threshold  value.  (DoDI  5000.2) 

OFEN-AIR  TEST  Testing  performed  in  an  outdoor  operating  environment,  i.e.,  on 
an  open-air  test  range  or  on  an  airborne  platform.  (EW  T&E  Task  Force) 

OPEN  LOOP  A  test  scenario  in  which  only  one  system  is  allowed  to  interact  to 
another's  actions.  For  example,  in  the  test  of  jamm»  against  a  SAM  system, 
emissions  from  the  SAM  are  received  by  the  jammer,  which  begins  jamming.  The 
simulated  SAM  system,  however,  is  not  allowed  to  receive  or  react  to  this  jamming. 
(EW  T&E  Task  Force) 

OPERATIONAL  ASSESSMENT  An  evaluation  of  operational  effectiveness  and 
operational  suitability  made  by  an  independent  operational  test  activity,  with  user 
suppon  as  required,  on  other  than  production  systems.  The  focus  of  an  operational 
assessment  is  on  significant  trends  noted  in  develoinnent  efforts,  programmatic  voids, 
areas  of  risk,  adequacy  of  requirements,  and  the  ability  of  the  program  to  support 
adequate  operatioi^  testing.  Operational  assessments  may  be  made  at  any  time  using 
technology  demonstrators,  prototypes,  mockups,  engineering  developnmnt  models,  or 
simulations  but  will  not  substitute  for  the  independent  operational  test  and  evaluation 
necessary  to  suppon  full  production  decisions.  (DoDI  5000.2) 

OPERATIONAL  CONCEPT  The  way  in  which  forces  and  equipment  are  arranged 
and  employed  in  battle.  This  includes  both  doctrine  and  tactics  concerning  how  a 
system  would  be  used  to  accomplish  national  objectives.  (DoDD  5000. 2-M) 

OPERATIONAL  EFFECTIVENESS  The  overall  degree  of  mission  accomplishment 
of  a  system  wtwn  used  by  representative  personnel  in  the  environment  planittd  or 
expected  (e.g.,  natural,  electronic,  threat,  etc.)  for  operational  employment  of  the 
system  considering  organization,  doctrine,  tactics,  survivability,  vulnerability,  and 
threat  (including  countermeasures,  initial  nuclear  weapons  effects,  nimlear,  biological, 
and  chemical  contamination  (NBCC)  threats).  (DoDI  5000.2) 

OPERATIONAL  REQUIREMENT  An  established  need  justifying  the  timely 
allocation  of  resources  to  achieve  a  capability  to  accomplish  approved  military 
objectives.  (Joint  Pub  1-02) 

OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD)  A  formatted  statement 
containing  performance  (operational  effectiveness  and  suitability)  and  related 


B-ll 


operational  parameters  for  the  proposed  concept  or  system.  It  is  to  develop 
requirements  for  contract  specific^ons  during  each  aa]uisition  phase.  The  ORD  is 
initially  prepared  by  the  user  or  user's  rquesentadve  during  phase  0,  ConcqM 
Esqiloratitm  and  Definition,  for  the  preferred  concq)t(s)  to  be  proposed  at  Milestone  1. 
(DoD  S000.2>M) 

OPERATIONAL  SUITABILITY  The  degree  to  which  a  system  can  be  placed 
satisfoctorily  in  field  use.  considering  availability,  compatibili^,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human  fimtors, 
maipower  scpportability,  logistics  sipportability,  natural  environmental  effects  and 
inpacts,  documentation,  and  training  requirements.  (DoDI  S000.2) 

OPERATIONAL  TEST  AGENCY  (OTA)  The  command  or  agency  designated  by 
the  program  management  directive,  or  other  tppropriate  directive,  as  responsible  for 
managing  and  conducting  the  independent  OT&E  of  a  system.  (DoDI  5()00.2) 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  Test  and  evaluation 
conducted  in  as  realistic  an  operational  environment  as  possible  to  determine  the 
operational  effectiveness  and  suitability  of  a  system  under  realistic  combat  conditions 
and  to  detennine  if  the  minimum  acceptable  operational  performance  requiren»nts  as 
specified  in  the  ORD  have  been  satisfied.  (DoDI  5(XX).2) 

OPERATIONS  SECURITY  (OPSEQ  A  process  of  identifying  critical  information 
and  subsequently  analyzing  friendly  actions  attendant  to  military  operations  and  other 
activities  to:  a.)  Identify  those  actions  that  can  be  observed  by  adversary  intelligence, 

b. )  Detennine  indicators  adversary  intelligence  might  obtain  that  could  be  interpreted 
or  pieced  together  to  derive  critical  information  in  time  to  be  useful  to  adversaries,  and 

c. )  Select  and  execute  measures  that  eliminate  or  reduce  to  an  accq>table  level  the 
vulnerabilities  of  friendly  actions  to  adversary  exploitation.  Also  called  OPSEC. 

(aCS  MOP  30) 

PERFORMANCE  Those  operational  and  sujport  characteristics  of  the  system  that 
allow  it  to  effectively  and  efficiently  perform  its  assigned  mission  over  time.  The 
support  characteristics  of  the  system  include  both  supportability  aspects  of  the  design 
and  the  support  elements  necessary  for  system  operation.  (DoDI  5000.2) 

PERFORMANCE  OBJECTIVE  The  performance  parameter  value  beyond  the 
minimum  operational  requirement  that  could  have  a  beneficial  impact  on  achieved 
operational  capability.  (DoDI  5000.2) 

PERFORMANCE  THRESHOLD  The  performance  parameter  value  that  meets  the 
minimum  level  of  system  performance  that  will  satisfy  the  validated  mission  need. 

Also  known  as  the  minimum  acceptable  operational  requirement  (DoDI  5000.2) 
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PRELIMINARY  DESIGN  REVIEW  (FDR)  A  review  conducted  on  each 
configuration  item  to  evaluate  the  progress,  technical  adequacy,  and  risk  resolution  of 
the  selected  design  approach;  to  determine  its  conqmtibility  with  performanK  and 
engineering  requirements  of  the  development  specification;  and  to  establish  the 
existence  and  compatibility  of  the  physical  and  functional  interface  among  the  item  and 
other  items  of  equipment,  facilities,  conqniter  programs,  and  personnel.  Conducted 
during  Phase  I,  DEMA^AL  (for  prototypes),  and  Phase  n,  EMD  (DoDI  5000.2) 

PRODUCIBILITY  The  relative  ease  of  Tnanufactiiring  an  item  or  system.  The 
relative  ease  is  governed  by  the  characteristics  and  features  of  a  design  that  enable 
economical  fabrication,  assembly,  inspection,  and  testing  using  available  manu£u;turing 
techniques.  (DoDI  5000.2) 

PRODUCTION  The  conversion  of  raw  materials  into  products  and/or  conqx>nents 
thereof,  through  a  series  of  manufacturing  processes.  It  includes  functions  of 
production  engineering,  controlling,  quality  assiuance,  and  the  determination  of 
resources  requirements.  (Joint  Pub  1-02) 

PRODUCTION  BASELINE  The  Production  Baseline  will  contain  objectives  and 
thresholds  for  key  cost,  schedule  and  performance  parameters  which  have  been  updated 
from  the  development  baseline.  The  Production  Baseline  will  be  submitted  as  a  stand¬ 
alone  part  of  the  Milestone  m  documentation.  (DoDI  5000.2) 

PRODUCTION  PROTOTYPE  (PF)  A  final  model  of  a  design  before  the  pilot  unit  is 
approved  for  production.  It  should  be  highly  representative  of  final  equipment,  except 
that  the  exact  manufacturing  assembly  process  and  production  design  changes  may  not 
yet  be  used  or  incorporated.  It  is  suitable  for  complete  evaluation  of  its  electrical 
and/or  mechanical  form  and  may  be  in  a  brassboard  (technical  and  operational 
configuration.  (EW  T&E  Task  Force) 

PROGRAM  EXECUTIVE  OFFICER  (FEO)  A  military  or  civilian  official  who  has 
primary  responsibility  for  directing  several  acquisition  category  I  programs  and  for 
assigned  ACAT  H,  HI,  and  FV  programs.  (DoDI  5(XX).2) 

PROGRAM  MANAGER  (PM)  A  military  or  civilian  official  who  is  responsible  for 
managing  an  acquisition  program.  (DoDI  5000.2) 

PROTOTYPE  A  model  suitable  for  evaluation  of  design,  performance,  and 
production  potential.  (Joint  Pub  1-02) 

PSYCHOLOGICAL  OPERATIONS  (PSYOP)  Planned  operations  to  convey 
selected  information  and  indicators  to  foreign  audiences  to  influence  their  emotions, 
motive,  objective  reasoning,  and  ultimately  the  behavior  of  foreign  governments, 
organizations,  groups,  and  individuals.  The  purpose  of  PSYOP  is  to  induce  or 
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reinforce  fineign  attitude  and  bdiavior  Avorable  to  the  origiiiator's  (d>jectives.  (CJCS 
MOP  30) 

RELIABILITY  The  ability  of  a  system  and  its  pans  to  perfbnn  its  mission  widimit 
fidliue,  d^iadatiom  or  demand  on  the  suf^xxt  system  (DoDI  5000>2) 

RELIANCE  STUDY  The  "Reliance"  process  for  T&E  is  an  outgrowth  of  the  DoD 
Management  R^rt  Decision  of  November  1990  viiose  objectives  were  to  "right-size” 
the  T&E  infiBStmcnue  for  the  ftnnre,  to  eliminate  unwarranted  dt^lication  and  to 
count,  in  the  future,  on  inter-service  "rslianoe*  for  non-service-peculiar  T&E  support 
capabilities.  EW  T&E  is  a  key  elemem  of  foat  stntty  effort.  (Reliance  Study) 

REQUIREMENT  An  established  need  justifying  the  timely  allocadcm  of  resources  to 
achieve  a  capability  to  accomplish  ^iproved  military  objectives,  missions,  or  tasks. 
(Joim  Pub  1-02) 

REQUIREMENTS  CORREXATION  MATRIX  (RCM)  A  management  tool  used  to 
provide  a  system  audit  trail.  It  contains  a  comparison  of  the  user's  needs,  system 
requirements,  contractual  specifications,  and  operational  evaluation  criteria.  (EW  T&E 
Task  Force) 

RISK  A  subjective  assessment  made  regarding  the  likelihood  or  probability  of  not 
achieving  a  specific  objective  by  the  time  established  with  the  resources  provided  or 
requested.  It  also  refers  to  overall  program  risk.  (DoDI  5(XX).2) 

SELF-PROTECTION  EW  systems  which  are  integrated  or  carried  on  board  a  host 
platform  and  provide  platform  self-protection  through  active  transmission  or  reflection 
of  electromagnetic  energy  or  destruction  of  enemy  command,  control,  and 
communications  systems.  (DoD  EW  PLAN) 

SIMULATION  Simulation  is  a  method  for  inylementing  a  model.  It  is  the  process  of 
conducting  experiments  with  a  model  for  the  lairpose  of  understanding  the  behavior  of 
the  system  modeled  under  selected  conditions  or  of  evaluating  various  strategies  for  the 
operation  of  the  system  within  the  limits  imposed  by  developmental  or  operational 
criteria.  (DoDI  5(XX).2) 

SPECIFICATIONS  Contractual  values  that  mflect  the  expected  capabilities  to  be 
produced  and/or  fielded  and  that  ate  traceable  to  the  cost,  schedule,  and  performance 
objectives  of  the  acquisition  program  baseline.  They  are  also  tied  to  ttte  acquisition 
phase  in  which  the  program  is  currently  engaged  and  reflect  the  demonstration 
requirements  is  support  of  exit  criteria.  (DoDI  5000.2) 

SPECTRUM  MANAGEMENT  Planning,  coordinating,  and  managing  joint  use  of  the 
electromagnetic  spectrum  through  operational,  engineering,  and  administrative 
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procedures,  with  the  objective  of  aiid>ltiig  eJectronic  systems  to  perform  dieir  fonctions 
in  the  intended  environment  widiout  causing  or  suffering  unaccquable  interferCTce. 
(CTCS  MOP  6) 

SUFPORTABILITY  The  degree  to  which  system  design  charactnistics  and  planned 
logistics  resources,  mett  system  peacetime  readiness  and  wartime  utilization 
requiremmits.  (DoDI  S000.2) 

SUPPRESSION  OF  ENEMY  AIR  DEFENSES  (SEAD)  That  activity  which 
neutralizes,  destroys,  or  tempmariiy  d^rades  enemy  air  defenses  in  a  specific  area  by 
pli^ical  attack,  deception,  and/or  electronic  warfere.  (CJCS  MOP  (5) 

SURGE  An  increase  in  the  production  or  rqtair  of  defense  goods  of  limited  duration. 
(DoDI  5000.2) 

SURVIVABILITY  The  capability  of  a  system  to  avoid  or  withstand  man-made  hostile 
environments  without  suffering  an  abortive  impairment  of  its  abiliQr  to  accomplish  its 
designated  mission.  (DoDI  S(X)0.2) 

SUSCEPTIBILITY  The  degree  to  which  a  device,  equipment,  or  weapon  system  is 
open  to  effective  attack  due  to  one  or  nmre  inherent  weakness.  Susceptibility  is  a 
fiinction  of  operational  tactics,  countermeasures,  probability  of  enemy  fielding  a  threat, 
etc.  Susceptibility  is  considered  a  subset  of  survii^ility.  (DoDI  SOOO-2) 

SYNTHESIS  In  the  T&E  Process  usage,  synthesis  is  the  combining  and  examining  of 
processed  information  (Step  Three)  with  expected  outcomes  (Step  Two)  and  other 
infonnation  using  both  technical  and  operational  judgement.  (EW  T&E  Task  Force) 

SYSTEM  Any  organized  assembly  of  resources  and  procedures  united  and  regulated 
by  interaction  or  interdependence  to  accomplish  a  set  of  specific  functions  (Joint  Pub 
1-02) 

SYSTEM  EFFECTIVENESS  A  measure  of  the  extent  to  which  a  system  may  be 
expected  to  achieve  a  set  of  specific  mission  requirements  expressed  as  a  function  of 
availability,  dependability,  and  capability.  (EW  T&E  Task  Force) 

SYSTEM  MATURITY  MATRIX  An  acquisition  raanagemem  tool  that  can  be  used 
to  highlight  differences  between  the  requir^  objective  and/or  threshold  values  aiui  the 
demonstrated  values  resulting  from  scheduled  testing.  (EW  T&E  Task  Force) 

SYSTEM  PERFORMANCE  PARAMETER  (SPP)  A  measurable  property  of  a 
system  cotKcpt,  a  system  design,  or  a  system  configuration  that  characterizes  system 
performance,  e.g.,  speed  in  knots  or  transmitted  power  in  watts,  that  can  be  directly 
measured  by  instrumenting  the  system  under  tKt.  (EW  T&E  Task  Force) 
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SYSTEM REUABIUTY AND MAlNTAINABnJTY PARAMETER  Ameasuie 
of  reliability  or  maintainability  in  iniiich  the  units  of  measurement  are  directly  related 
to  operational  readiness,  mission  success,  maintenance  manpower  or  logistic  support 
cost.  (DoDI  S000.2) 

SYSTEM  THREAT  ASSESSMENT  A  systons-specific  assessmnit  that  desmibes 
the  threat  to  be  countered  and  the  projected  threat  environment.  The  threat  assessment 
wUl  be  derived  ftom  DIA  produced  or  validated  documents.  (DoDI  5000.2) 

SYSTEMS  ENGINEERING  A  comprehensive,  iterative  technical  management 
process  to:  1)  Translate  an  opnadonal  need  into  a  configured  system  meeting  that  need 
through  a  systematic,  concurrent  pproach  to  integrated  design  of  the  systmn  and  its 
related  manufacturing,  test,  and  sipport  processes;  2)  Integrate  the  technical  inputs  of 
the  entire  development  community  and  all  technical  disciplines  into  a  coordinated  effort 
that  meets  established  program  cost,  schedule,  and  performance  objectives;  3)  Ensure 
the  compatibility  of  all  functional  and  physical  interfaces  and  ensure  that  system 
definition  and  design  reflect  the  requirements  for  all  system  elements;  4)  Characterize 
ti^rhnifai  rislcs.  develop  risk  abatement  aiproaches,  and  reduce  technical  risk  through 
early  test  and  demonstration  of  system  elements.  (DoDI  5000.2) 

TECHNICAL  DATA  Scientific  or  technical  information  recorded  in  any  form  or 
medium  (such  as  manuals  and  drawings).  Conputer  programs  and  related  software  are 
not  technical  data;  documentation  of  computer  programs  and  related  software  are.  Also 
excluded  are  financial  data  or  other  information  related  to  contract  administration. 
(DoDI  5000.2) 

TECHNICAL  PERFORMANCE  PARAMETER  (TPP)  A  selected  subset  of  system- 
specific  performance  parameters  used  as  the  technical  measures  tracked  in  the  systems 
engineering  technical  performance  measurement  program.  TPPs  are  used  to  measure 
compliance  with  requirements  and  to  assess  the  level  of  technical  risk  in  a  development 
program.  (EW  T&E  Task  Force) 

TECHNICAL  REQUIREMENTS  FORMULATION  The  process  of  converting 
operational  requirements  into  technical  requirements  that  can  be  acted  on  by  designers. 
(DSMC) 

TEST  AND  EVALUATION  (T&E)  Any  project  or  program  designed  to  obtain, 
verify,  and  provide  data  for  the  evaluation  of  research  and  development  other  than 
laboratory  experiments  for  the  purpose  of  determining  if  the  minimum  acceptable 
operational  performance  requirements  as  specified  in  the  Operational  Requirements 
Document  have  been  satisfied.  (DoDI  5(XX).2) 
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TEST  AND  EVALUATION  MASTER  FLAN  (TEMP)  An  ovendl  T&E  plan 
designed  to  idendfy  and  integrate  the  efforts  and  scfaedutes  of  all  T&E  to  be  done  in  an 
acquisition  program.  (DoDl  5000.2) 

TESTBED  A  system  representation  consisting  partially  of  artiiai  hardware  and/or 
software  and  partially  of  conqniter  models  or  prototype  hardware  and/or  software. 
(DoDI  5000.2) 

TEST  CONDITIONS  The  environment  (e.g.,  location*  weather),  scenario,  and 
operating  procedures  and  configurations  for  the  SUT  and  adversaries  in  the  test 
scenario.  (EW  T&E  Task  Force) 

TEST  ENVIRONMENT  Hie  test  location,  facility  type,  weadier  conditions,  threat, 
electromagnetic  and  stimulation  environments,  etc.,  im^  which  the  test  is  conducted. 
(EW  T&E  Task  Force) 

TEST  EVENT  An  activity  during  conduct  of  a  test  trial  that  requires  a  response  by 
the  system  and/or  personnel  under  test.  (EW  T&E  Task  Force) 

TEST  OBJECTIVE  The  specific  performance  or  technical  parameters  to  be  measured 
during  the  test  to  evaluate  system  performance,  system  operational  effectiveness,  or 
system  suitability.  (EW  T^  Task  Force) 

TEST  RESOURCES  A  collective  term  that  encompasses  all  elements  necessary  to 
plan,  conduct,  and  collect/analyze  data  from  a  test  evem  or  program.  (AR  73-1) 

TEST  SCENARIO  A  situation,  representative  of  what  the  system  under  test  may 
encounter  in  real  life,  that  is  used  to  enact  a  set  of  events  between  it  and  adversaries 
included  in  the  situation  (e.g.,  threat  simulator  locations  and  flight  profiles).  (EW  T&E 
Task  Force) 

TESTBED  A  system  representation  consisting  partially  of  actual  hardware  and/or 
software  and  partially  of  computer  models  or  prototype  hardware  and/or  software. 
(DoDI  5000.2) 

THREAT  Currem  and  future  capabilities  of  a  potential  enemy  force  against  one  or 
more  US  developmental  systems  in  terms  of  combat  materiel,  employment,  doctrine, 
force  structure,  and  combat  enviroiunent.  (AR  73-1) 

THRESHOLD  A  minimum  acceptable  value  for  a  performance  parameter  which,  in 
the  user's  judgment,  is  necessary  to  provide  an  operational  capability  that  will  satisfy 
the  mission  need.  The  threshold  must  be  met  in  order  to  gain  approval  from  the 
milestone  decision  authority.  (DoDI  5000.2) 
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UPGRADE  A  change  to  a  system  (nether  for  safe^,  to  correct  a  de&imx^,  or  to 
iiiq)rove  program  performance)  to  a  system  that  is  out  of  produmion.  Upgrades  ate  part 
of  the  Milestone  0  decision  process.  (DoDI  5000.2) 

VALIDATION  The  process  of  determining  the  degree  to  which  a  model  is  an  accurate 
representation  of  the  real-world  from  the  perspective  of  the  intended  uses  of  the  model. 
(DoDD  5000.59) 

VALUE  An  assigned  or  calculated  numerical  quantity.  An  amount  conskfered  u>  be  a 
suitable  etpiivalent  for  somediing  else.  (AMERICAN  HERITAGE  DICTIONARY) 

VERIFICATION  The  process  of  determining  that  a  model  inq>lemBntation  accurately 
represents  the  developers'  conceptual  description  and  specifications.  (DoDD  5000.59) 

VETRONICS  Electrical  and  electronic  systems  aboard  ground  vehicles.  (Commonly 
used  military  jargon  that  is  a  contraction  of  j^hicle  electronics)  (EW  T&E  Task  Force) 

VULNERABILITY  The  characteristics  of  a  system  that  cause  it  to  suffer  a  defrnite 
degradation  (loss  or  reduction  of  capabiliQr  to  perform  the  designated  mission)  as  a 
result  of  having  been  subjected  to  a  certain  (ctefined)  level  of  effects  in  an  unnanual 
(man-made)  hostile  environment.  Vulnerability  is  considered  a  subset  of  survivability. 
(DoDI  5000.2) 

WARTIME  RESERVE  MODES  (WARM)  Characteristics  and  operating  procedures 
of  sensor,  communications,  navigation  aids,  threat  recognition,  we^ns,  and 
countermeasures  systems  that  (a)  will  contribute  to  military  effectiveness  if  unknown  to 
or  misunderstood  by  opposing  commanders  before  they  are  used,  but  (b)  could  be 
exploited  or  neutralized  if  known  in  advance.  Wartime  reserve  modes  are  deliberately 
held  in  reserve  for  wartime  or  emergency  use  and  seldom,  if  ever,  aR)iied  or 
intercepted  prior  to  such  use.  (CJCS  MOP  6) 
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